Oct 30 05:28:56.307602 ip-10-0-138-15 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 05:28:56.307613 ip-10-0-138-15 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 05:28:56.307620 ip-10-0-138-15 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 05:28:56.307841 ip-10-0-138-15 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 05:29:06.396354 ip-10-0-138-15 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 05:29:06.396374 ip-10-0-138-15 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ef338ee24cdf4e7a92d65fd8736a1f07 -- Oct 30 05:31:12.651212 ip-10-0-138-15 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:31:13.047542 ip-10-0-138-15 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:13.047542 ip-10-0-138-15 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:31:13.047542 ip-10-0-138-15 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:13.047542 ip-10-0-138-15 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:13.051017 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.050899 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.051017 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051016 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.051017 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051021 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051024 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051026 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051030 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051032 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051035 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051037 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051040 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051043 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051046 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051048 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051051 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051053 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051057 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051060 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051062 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051065 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051068 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051070 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.051113 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051073 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051076 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051078 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051085 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051088 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051091 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051093 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051096 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051099 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051101 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051104 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051106 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051109 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051111 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051114 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051117 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051120 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051123 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051126 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051128 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.051565 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051131 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051134 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051137 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051139 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051142 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051144 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051146 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051149 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051151 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051154 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051156 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051159 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051161 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051165 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051170 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051173 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051176 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051179 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051182 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051184 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.052077 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051186 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051189 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051192 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051194 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051197 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051200 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051203 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051207 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051209 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051212 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051215 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051218 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051221 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051223 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051226 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051228 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051231 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051233 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051236 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.052544 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051238 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051242 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051245 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051249 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051252 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051254 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051257 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051260 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051263 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051265 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051268 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051271 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051274 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051277 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051279 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051282 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051285 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051287 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051289 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051292 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.053010 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051295 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051298 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051301 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051305 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051307 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.051311 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052137 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052145 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052147 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052150 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052153 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052156 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052159 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052161 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052164 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052167 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052169 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052172 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052174 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.053481 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052177 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052181 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052185 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052188 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052191 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052195 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052199 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052203 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052206 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052209 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052212 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052214 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052217 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052220 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052222 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052225 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052227 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052230 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052233 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052236 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.053956 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052239 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052241 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052244 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052247 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052249 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052251 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052254 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052256 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052259 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052262 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052265 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052267 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052270 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052273 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052275 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052278 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052281 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052283 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052286 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.054430 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052288 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052290 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052292 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052295 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052297 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052300 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052302 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052305 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052308 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052310 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052313 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052315 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052317 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052321 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052324 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052327 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052329 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052332 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052335 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052337 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.054878 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052340 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052343 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052346 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052348 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052351 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052354 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052356 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052359 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052361 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052364 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052367 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052369 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052372 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052374 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052376 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052379 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052382 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052384 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052387 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052389 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.055363 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052392 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052395 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052397 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052400 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052402 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052405 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052408 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052413 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052416 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052418 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052421 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052423 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052426 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052429 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052984 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052990 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052993 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052996 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.052999 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.055853 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053002 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053004 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053007 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053010 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053012 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053015 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053017 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053020 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053023 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053026 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053028 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053031 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053034 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053036 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053039 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053042 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053045 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053048 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053050 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053058 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.056312 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053064 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053067 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053070 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053072 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053075 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053078 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053080 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053083 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053085 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053088 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053090 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053093 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053096 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053098 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053101 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053103 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053106 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053108 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053111 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.056783 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053113 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053117 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053121 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053124 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053128 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053130 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053133 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053135 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053138 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053140 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053143 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053145 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053148 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053157 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053160 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053162 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053165 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053167 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053170 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.057261 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053172 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053175 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053177 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053179 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053182 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053184 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053187 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053189 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053192 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053194 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053196 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053200 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053202 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053205 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053207 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053210 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053212 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053215 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053218 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053220 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.057706 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053223 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053225 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053227 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053230 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053232 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053235 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053240 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053248 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053251 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053253 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053257 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053261 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053264 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053267 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053269 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053272 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053274 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053277 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053280 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.058190 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053283 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053286 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053288 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.053290 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053385 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053393 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053404 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053409 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053413 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053416 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053422 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053426 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053429 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053432 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053436 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053440 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053443 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053446 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053449 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053453 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053456 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053465 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053472 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:31:13.058677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053474 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053477 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053481 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053484 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053488 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053492 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053495 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053498 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053501 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053504 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053507 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053510 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053513 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053517 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053520 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053523 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053526 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053529 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053532 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053541 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053544 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053547 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053551 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053554 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:31:13.059231 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053558 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053561 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053564 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053567 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053570 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053574 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053577 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053585 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053588 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053591 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053594 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053598 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053601 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053604 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053607 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053610 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053613 2583 flags.go:64] FLAG: --help="false" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053616 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053619 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053622 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053625 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053628 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053631 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053634 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:31:13.059803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053636 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053639 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053642 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053650 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053653 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053656 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053659 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053662 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053665 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053668 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053670 2583 flags.go:64] FLAG: --lock-file="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053673 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053676 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053679 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053685 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053688 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053697 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053700 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053703 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053706 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053709 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053712 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053717 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053720 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053724 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 05:31:13.060420 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053727 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053730 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053733 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053736 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053739 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053742 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053744 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053751 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053754 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053757 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053762 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053765 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053767 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053770 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053773 2583 flags.go:64] FLAG: --port="10250" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053776 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053779 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00017c86398f70049" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053782 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053785 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053788 2583 flags.go:64] FLAG: --register-node="true" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053791 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053795 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053797 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053800 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053809 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053813 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:31:13.061051 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053816 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053818 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053821 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053824 2583 flags.go:64] FLAG: --runonce="false" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053827 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053830 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053833 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053836 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053838 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053841 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053845 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053848 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053850 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053853 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053856 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053859 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053863 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053866 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053869 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053874 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053877 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053900 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053908 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053911 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053913 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:31:13.061686 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053916 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053919 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053922 2583 flags.go:64] FLAG: --v="2" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053926 2583 flags.go:64] FLAG: --version="false" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053930 2583 flags.go:64] FLAG: --vmodule="" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053934 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.053943 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054128 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054132 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054135 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054138 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054142 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054144 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054147 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054150 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054152 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054155 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054157 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054160 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054162 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054165 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.062278 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054167 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054169 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054173 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054176 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054178 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054181 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054184 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054186 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054189 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054191 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054194 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054197 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054200 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054203 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054206 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054208 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054211 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054213 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054217 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054219 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.062794 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054222 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054224 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054227 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054229 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054232 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054234 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054237 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054240 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054242 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054245 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054247 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054249 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054252 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054254 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054258 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054261 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054263 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054266 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054268 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.063295 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054271 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054273 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054276 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054278 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054282 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054285 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054287 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054290 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054292 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054295 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054297 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054300 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054303 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054305 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054308 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054310 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054313 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054315 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054319 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.063749 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054323 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054326 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054328 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054331 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054333 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054336 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054338 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054341 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054345 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054347 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054350 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054353 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054355 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054358 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054361 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054363 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054366 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054369 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054372 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054375 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.064206 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054377 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054380 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054383 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054385 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054388 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054390 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054395 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054398 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054401 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054404 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054407 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054409 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054412 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054414 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.054441 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:13.064679 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054486 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054490 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054493 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054495 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054498 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054501 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054504 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054508 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054522 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054525 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054528 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054531 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054533 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054537 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054539 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054543 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054546 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054548 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054551 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.065054 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054554 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054556 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054559 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054561 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054564 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054567 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054570 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054572 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054575 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054577 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054580 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054582 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054585 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054587 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054590 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054592 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054595 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054597 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054601 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054604 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.065504 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054607 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054609 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054612 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054615 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054617 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054620 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054622 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054625 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054629 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054631 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054634 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054636 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054639 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054641 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054644 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054646 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054649 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054651 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054654 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054657 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.065978 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054659 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054662 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054664 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054667 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054669 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054671 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054674 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054676 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054679 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054681 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054685 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054687 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054690 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054693 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054695 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054698 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054700 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054703 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054705 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054707 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.066494 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054711 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054714 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054716 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054719 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054721 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054724 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054726 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054730 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054733 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054736 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054739 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054742 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054744 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054747 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054749 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054752 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054754 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054757 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054759 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.067063 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054762 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054764 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054767 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054770 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054773 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054775 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054778 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.054781 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.062242 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.062260 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.062929 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.062936 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063020 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063026 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063029 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063032 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063036 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063039 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063041 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063044 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.067550 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063046 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063049 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063051 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063055 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063068 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063071 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063074 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063076 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063079 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063082 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063084 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063086 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063089 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063092 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063095 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063098 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063100 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063102 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063105 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.068051 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063108 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063111 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063114 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063116 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063119 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063122 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063125 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063128 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063130 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063132 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063135 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063137 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063140 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063142 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063145 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063148 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063150 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063153 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063160 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.068501 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063163 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063165 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063168 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063170 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063173 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063175 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063178 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063182 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063184 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063187 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063189 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063192 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063194 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063198 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063203 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063206 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063209 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063211 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063214 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.069002 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063216 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063218 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063221 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063224 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063226 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063228 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063231 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063234 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063236 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063239 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063241 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063244 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063246 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063254 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063257 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063259 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063261 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063264 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063266 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063269 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.069463 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063271 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063274 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063276 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063279 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063281 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063284 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063286 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063289 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063292 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063295 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063298 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063300 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063303 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063306 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063308 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063311 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063313 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063316 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063318 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063321 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.069965 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063323 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.063354 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063547 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063551 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063555 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063558 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063567 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063570 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063573 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063577 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063579 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063582 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063585 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063587 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063590 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063592 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.070422 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063594 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063597 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063599 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063602 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063604 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063607 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063610 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063613 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063615 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063617 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063620 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063622 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063625 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063628 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063630 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063633 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063635 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063638 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063640 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063642 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.070850 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063645 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063647 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063650 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063658 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063660 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063663 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063665 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063668 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063671 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063673 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063676 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063679 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063682 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063684 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063686 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063689 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063691 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063694 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063696 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063699 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.071326 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063701 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063703 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063707 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063710 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063713 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063716 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063718 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063721 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063724 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063726 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063728 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063731 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063733 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063736 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063738 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063740 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063748 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063751 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.071790 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063754 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063758 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063762 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063765 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063769 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063772 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063774 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063777 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063779 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063788 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063791 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063793 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063796 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063798 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063801 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063803 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063806 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063808 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063811 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.072217 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063813 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063816 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063818 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063821 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063823 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063826 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063828 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063831 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063833 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063835 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063838 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063840 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063848 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063851 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063854 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.063879 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:13.073381 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063948 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063953 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063956 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063959 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063962 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063965 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063968 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063970 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063973 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063975 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063978 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063980 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063982 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063985 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063987 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063990 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063992 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063994 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063997 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:13.074003 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.063999 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064002 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064004 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064007 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064009 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064012 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064014 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064017 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064019 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064022 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064029 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064032 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064037 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064039 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064041 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064044 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064047 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064049 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064052 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064054 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:13.074484 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064057 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064059 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064062 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064064 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064067 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064069 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064071 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064074 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064076 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064079 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064081 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064084 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064086 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064088 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064091 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064093 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064096 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064098 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064101 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064103 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:13.075005 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064106 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064108 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064111 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064119 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064123 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064125 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064127 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064130 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064132 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064135 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064137 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064141 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064144 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064147 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064149 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064151 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064154 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064157 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064160 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:13.075470 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064162 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064164 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064167 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064169 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064172 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064174 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064176 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064179 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064181 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064185 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064192 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064195 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064197 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064200 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064202 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064205 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064208 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064217 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064220 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:13.075917 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064223 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064225 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064228 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064231 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064233 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064236 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064238 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064240 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:13.064243 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.064300 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.066532 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.067484 2583 server.go:1014] "Starting client certificate rotation" Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.067612 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.067649 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.069512 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:31:13.076377 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.074912 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:31:13.084807 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.084787 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:31:13.088390 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.088359 2583 log.go:25] "Validated CRI v1 image API" Oct 30 05:31:13.089319 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.089296 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:31:13.096131 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.096102 2583 fs.go:135] Filesystem UUIDs: map[3f8b49d7-d447-46f1-a9f6-54085d4679fc:/dev/nvme0n1p4 7734fa11-071d-4d77-a8b9-b32636076296:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 05:31:13.096219 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.096129 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:31:13.101483 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.101362 2583 manager.go:217] Machine: {Timestamp:2026-10-30 05:31:13.100037335 +0000 UTC m=+0.333248806 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099126 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ef2528acf69accb60f72c6adef420 SystemUUID:ec2ef252-8acf-69ac-cb60-f72c6adef420 BootID:ef338ee2-4cdf-4e7a-92d6-5fd8736a1f07 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:1d:ee:cf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:1d:ee:cf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:de:2e:72:91:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:31:13.101554 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.101486 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:31:13.101630 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.101619 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:31:13.104089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.104059 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:31:13.104237 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.104091 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:31:13.104282 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.104244 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:31:13.104282 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.104252 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:31:13.104338 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.104288 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:31:13.104338 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.104309 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:31:13.105172 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.105161 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:31:13.109229 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.109216 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:31:13.109359 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.109350 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:31:13.110112 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.110102 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:31:13.110161 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.110116 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:31:13.110161 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.110133 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:31:13.110161 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.110148 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:31:13.110161 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.110161 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:31:13.110565 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.110549 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:31:13.112947 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.112925 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:31:13.118027 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.118015 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:31:13.118865 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.118852 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:31:13.118932 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.118903 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:31:13.119067 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119053 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:31:13.119105 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119081 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:31:13.119105 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119092 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:31:13.119105 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119101 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:31:13.119181 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119110 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:31:13.119181 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119120 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:31:13.119556 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119547 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:31:13.119584 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119557 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:31:13.119584 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119564 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:31:13.119584 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119570 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:31:13.119584 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119578 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:31:13.119700 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.119587 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:31:13.120259 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.120250 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:31:13.120291 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.120260 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:31:13.125699 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.125680 2583 server.go:1263] "Started kubelet" Oct 30 05:31:13.125802 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.125755 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:31:13.125926 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.125858 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:31:13.125966 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.125953 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:31:13.126565 ip-10-0-138-15 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:31:13.127215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.127144 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:31:13.127942 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.127926 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:31:13.134108 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.134088 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:31:13.134837 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.134823 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:31:13.134919 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.134838 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:31:13.134919 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.134909 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.135533 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135517 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:31:13.135684 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135517 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:31:13.135756 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135689 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:31:13.135756 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135642 2583 factory.go:55] Registering systemd factory Oct 30 05:31:13.135756 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135718 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:31:13.135908 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135789 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:31:13.135908 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.135799 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:31:13.136024 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.136013 2583 factory.go:153] Registering CRI-O factory Oct 30 05:31:13.136067 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.136027 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 05:31:13.136099 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.136084 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:31:13.136130 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.136108 2583 factory.go:103] Registering Raw factory Oct 30 05:31:13.136130 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.136127 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 05:31:13.137224 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.137113 2583 manager.go:319] Starting recovery of all containers Oct 30 05:31:13.148851 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.148832 2583 manager.go:324] Recovery completed Oct 30 05:31:13.149366 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.149333 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:31:13.154456 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.154438 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.154852 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.154833 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-znm8t" Oct 30 05:31:13.156274 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156257 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.156348 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156292 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.156348 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156307 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.156927 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156912 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:31:13.156927 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156926 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:31:13.157054 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156947 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:31:13.157054 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.156955 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.157797 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.157775 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:31:13.158341 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.158323 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.158422 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.158354 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.158422 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.158368 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.159249 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.159233 2583 policy_none.go:50] "Start" Oct 30 05:31:13.159249 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.159250 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:31:13.159352 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.159260 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:31:13.161204 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.161193 2583 policy_none.go:44] "Start" Oct 30 05:31:13.168007 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.167828 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:31:13.168878 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.167827 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-15.ec2.internal.18e337a86bc4239a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-15.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-15.ec2.internal,},FirstTimestamp:2026-10-30 05:31:13.125639066 +0000 UTC m=+0.358850537,LastTimestamp:2026-10-30 05:31:13.125639066 +0000 UTC m=+0.358850537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-15.ec2.internal,}" Oct 30 05:31:13.172161 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.172137 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-znm8t" Oct 30 05:31:13.173229 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.173176 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:31:13.173320 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.173249 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:31:13.173392 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.173339 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:31:13.191578 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.191493 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-15.ec2.internal.18e337a86d9798d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-15.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-15.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-15.ec2.internal,},FirstTimestamp:2026-10-30 05:31:13.156274387 +0000 UTC m=+0.389485863,LastTimestamp:2026-10-30 05:31:13.156274387 +0000 UTC m=+0.389485863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-15.ec2.internal,}" Oct 30 05:31:13.195485 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.195455 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:31:13.198232 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198217 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 05:31:13.198316 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.198245 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:31:13.198316 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198254 2583 server.go:87] "Starting device plugin registration server" Oct 30 05:31:13.198506 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198496 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:31:13.198546 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198508 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:31:13.198660 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198628 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:31:13.198751 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198719 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:31:13.198751 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.198728 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:31:13.199306 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.199286 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:31:13.199374 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.199335 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.268361 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.268325 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:31:13.269990 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.269674 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:31:13.269990 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.269696 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:31:13.269990 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.269725 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:31:13.269990 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.269763 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:31:13.288373 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.288345 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:13.298870 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.298802 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.299796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.299778 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.299915 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.299810 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.299915 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.299826 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.299915 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.299862 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.320680 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.320651 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.320820 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.320685 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-15.ec2.internal\": node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.362947 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.362920 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.369983 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.369958 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal"] Oct 30 05:31:13.370088 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.370022 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.371528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.371512 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.371586 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.371545 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.371586 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.371555 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.373348 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.373335 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.373511 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.373497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.373555 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.373527 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.374203 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.374180 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.374312 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.374214 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.374312 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.374180 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.374312 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.374232 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.374312 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.374251 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.374312 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.374268 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.376133 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.376117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.376220 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.376147 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:13.376914 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.376880 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:13.376991 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.376926 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:13.376991 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.376935 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:13.405216 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.405197 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-15.ec2.internal\" not found" node="ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.411510 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.411491 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-15.ec2.internal\" not found" node="ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.437451 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.437425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddf44eee240eccf9696b703c5918959-config\") pod \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" (UID: \"eddf44eee240eccf9696b703c5918959\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.437592 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.437466 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.437592 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.437497 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.463712 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.463681 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.537634 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.537609 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddf44eee240eccf9696b703c5918959-config\") pod \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" (UID: \"eddf44eee240eccf9696b703c5918959\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.537812 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.537638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.537812 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.537658 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.537812 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.537700 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.537812 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.537729 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.537812 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.537729 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddf44eee240eccf9696b703c5918959-config\") pod \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" (UID: \"eddf44eee240eccf9696b703c5918959\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.564464 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.564405 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.665186 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.665156 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.708870 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.708844 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.715276 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:13.715256 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:13.766298 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.766264 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.866929 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.866832 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:13.967349 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:13.967316 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:14.066918 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.066870 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:31:14.067676 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.067089 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:31:14.067965 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:14.067942 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:14.168937 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:14.168860 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:14.177566 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.177533 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:26:14 +0000 UTC" deadline="2028-06-11 20:57:41.135395727 +0000 UTC" Oct 30 05:31:14.177566 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.177564 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14175h26m26.957836539s" Oct 30 05:31:14.182524 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:14.182484 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddf44eee240eccf9696b703c5918959.slice/crio-743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8 WatchSource:0}: Error finding container 743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8: Status 404 returned error can't find the container with id 743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8 Oct 30 05:31:14.183150 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:14.183127 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d26ba6ff099574f4393ea9d38dc6f8.slice/crio-cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6 WatchSource:0}: Error finding container cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6: Status 404 returned error can't find the container with id cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6 Oct 30 05:31:14.188361 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.188345 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:31:14.230217 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.230186 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:14.269305 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:14.269258 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:31:14.273746 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.273704 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" event={"ID":"eddf44eee240eccf9696b703c5918959","Type":"ContainerStarted","Data":"743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8"} Oct 30 05:31:14.274815 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.274793 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" event={"ID":"a5d26ba6ff099574f4393ea9d38dc6f8","Type":"ContainerStarted","Data":"cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6"} Oct 30 05:31:14.294284 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.294256 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:14.335390 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.335363 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:31:14.361805 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.361779 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:14.363779 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.363760 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:31:14.387597 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.387575 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:14.464453 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:14.464371 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:16.525578 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.525546 2583 apiserver.go:52] "Watching apiserver" Oct 30 05:31:16.550357 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.550327 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:31:16.572095 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.572064 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:31:16.572486 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.572457 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-kcw92","openshift-multus/network-metrics-daemon-4rx42","openshift-network-diagnostics/network-check-target-phtr7","kube-system/global-pull-secret-syncer-wb5rz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z","openshift-ovn-kubernetes/ovnkube-node-cr5xj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal","openshift-multus/multus-additional-cni-plugins-pcx4f","openshift-network-operator/iptables-alerter-lp49f","kube-system/konnectivity-agent-sxrx6","kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fzbh9","openshift-dns/node-resolver-q2qpx","openshift-image-registry/node-ca-8mh7z"] Oct 30 05:31:16.575222 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.575190 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.577410 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.577388 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.579803 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.579707 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:16.579803 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.579793 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:16.582306 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.582138 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.582306 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.582242 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:16.584222 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.584205 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.586550 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.586533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.590498 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.590478 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.592639 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.592364 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:31:16.592639 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.592484 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:31:16.592639 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.592491 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2ztbt\"" Oct 30 05:31:16.592884 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.592868 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.595564 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.595537 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.596538 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.596521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:16.596624 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.596586 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:16.598527 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.598506 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.601612 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.601592 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.602548 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.602521 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:31:16.602639 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.602598 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xp45m" Oct 30 05:31:16.605927 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.605210 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.607947 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.607928 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xp45m" Oct 30 05:31:16.651660 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.651634 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:31:16.657677 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.657653 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:16.662624 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662594 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-config\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.662748 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662688 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-var-lib-kubelet\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.662748 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b010236-561c-4748-842a-648d3290ff2b-agent-certs\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-cnibin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662770 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-conf-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662794 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-ovn\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-bin\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-hosts-file\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.662860 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662857 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-socket-dir-parent\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662870 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-kubelet\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662886 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-systemd\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-run\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662930 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-device-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662944 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-log-socket\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662973 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-script-lib\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.662994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-host\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eddf108a-bf5f-483b-ac39-1804ea6d8d76-host-slash\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-netns\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-bin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-dbus\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663067 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9766570-5a16-4a4a-aa4e-900bbbed459c-serviceca\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663081 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-registration-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-k8s-cni-cncf-io\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-multus\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-kubelet\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-socket-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663282 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-slash\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663297 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b010236-561c-4748-842a-648d3290ff2b-konnectivity-ca\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-tuned\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663336 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-tmp-dir\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxvws\" (UniqueName: \"kubernetes.io/projected/eddf108a-bf5f-483b-ac39-1804ea6d8d76-kube-api-access-sxvws\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663379 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663438 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szsn4\" (UniqueName: \"kubernetes.io/projected/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-kube-api-access-szsn4\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663463 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663484 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9766570-5a16-4a4a-aa4e-900bbbed459c-host\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663503 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663527 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4cwz\" (UniqueName: \"kubernetes.io/projected/17d8104e-e85c-490a-855d-63af3e241f11-kube-api-access-q4cwz\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-netns\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.663946 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663572 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663588 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-sys\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663611 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdfb\" (UniqueName: \"kubernetes.io/projected/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-kube-api-access-nvdfb\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663626 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-etc-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663640 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-netd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-lib-modules\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663709 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-kubelet-config\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663723 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663738 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-env-overrides\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663821 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-modprobe-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysconfig\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-kubernetes\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-tmp\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663886 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-multus-certs\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.664682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-systemd-units\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-systemd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663939 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.663995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eddf108a-bf5f-483b-ac39-1804ea6d8d76-iptables-alerter-script\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-cni-binary-copy\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vfz\" (UniqueName: \"kubernetes.io/projected/28230167-b221-4cb3-8601-fe8bd9605a73-kube-api-access-g9vfz\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-node-log\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664062 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-system-cni-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664132 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-conf\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664149 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bjx\" (UniqueName: \"kubernetes.io/projected/f9766570-5a16-4a4a-aa4e-900bbbed459c-kube-api-access-v7bjx\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664163 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovn-node-metrics-cert\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664190 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-cnibin\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664206 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-system-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.665426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-var-lib-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkwg\" (UniqueName: \"kubernetes.io/projected/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-kube-api-access-5lkwg\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664249 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-hostroot\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664263 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-sys-fs\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxfq8\" (UniqueName: \"kubernetes.io/projected/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-kube-api-access-pxfq8\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxk9\" (UniqueName: \"kubernetes.io/projected/0693da1f-8341-413f-b174-2575807a5a85-kube-api-access-4lxk9\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664309 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-multus-daemon-config\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.666163 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.664327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-etc-kubernetes\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.693005 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.692941 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:31:16.693005 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693000 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8j26\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693016 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693018 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xpx6p\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693000 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pqn7\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693192 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693193 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693203 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693220 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjhr7\"" Oct 30 05:31:16.693303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693300 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693576 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693608 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vg6rp\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693626 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jtd4z\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693637 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693627 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693656 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693713 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k22sc\"" Oct 30 05:31:16.693781 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693765 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693810 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693816 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693816 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf7fl\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693842 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693865 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693875 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693902 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693909 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693919 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693922 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693948 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693951 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.694089 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.693969 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:31:16.706816 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.706777 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:31:16.764798 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764767 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-multus\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.764798 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-kubelet\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764824 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-socket-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-slash\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b010236-561c-4748-842a-648d3290ff2b-konnectivity-ca\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764895 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-kubelet\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764904 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-tuned\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-tmp-dir\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764947 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-slash\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxvws\" (UniqueName: \"kubernetes.io/projected/eddf108a-bf5f-483b-ac39-1804ea6d8d76-kube-api-access-sxvws\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764979 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-socket-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.765038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.764895 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-multus\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765211 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szsn4\" (UniqueName: \"kubernetes.io/projected/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-kube-api-access-szsn4\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9766570-5a16-4a4a-aa4e-900bbbed459c-host\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765179 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765277 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765308 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765330 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cwz\" (UniqueName: \"kubernetes.io/projected/17d8104e-e85c-490a-855d-63af3e241f11-kube-api-access-q4cwz\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765341 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9766570-5a16-4a4a-aa4e-900bbbed459c-host\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-netns\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765374 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-tmp-dir\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765393 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-sys\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.765404 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdfb\" (UniqueName: \"kubernetes.io/projected/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-kube-api-access-nvdfb\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.765528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765439 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-etc-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.765451 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:17.265434537 +0000 UTC m=+3.084425430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765469 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-netd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765483 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-etc-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765514 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-netns\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-lib-modules\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765563 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-kubelet-config\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765597 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-env-overrides\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765604 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-modprobe-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b010236-561c-4748-842a-648d3290ff2b-konnectivity-ca\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765672 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-netd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765700 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-modprobe-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.765985 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysconfig\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-kubernetes\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-tmp\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-multus-certs\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765820 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-systemd-units\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-systemd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765853 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765875 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765900 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765925 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-sys\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765941 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eddf108a-bf5f-483b-ac39-1804ea6d8d76-iptables-alerter-script\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.765972 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.765992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-cni-binary-copy\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.766025 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:31:17.266009527 +0000 UTC m=+3.085000418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vfz\" (UniqueName: \"kubernetes.io/projected/28230167-b221-4cb3-8601-fe8bd9605a73-kube-api-access-g9vfz\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.766698 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-node-log\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766077 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-system-cni-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766134 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766159 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-conf\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bjx\" (UniqueName: \"kubernetes.io/projected/f9766570-5a16-4a4a-aa4e-900bbbed459c-kube-api-access-v7bjx\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovn-node-metrics-cert\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766240 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-cnibin\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-system-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766290 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-var-lib-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkwg\" (UniqueName: \"kubernetes.io/projected/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-kube-api-access-5lkwg\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-hostroot\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-sys-fs\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766428 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfq8\" (UniqueName: \"kubernetes.io/projected/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-kube-api-access-pxfq8\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766456 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxk9\" (UniqueName: \"kubernetes.io/projected/0693da1f-8341-413f-b174-2575807a5a85-kube-api-access-4lxk9\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766467 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-kubernetes\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-multus-daemon-config\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767358 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-etc-kubernetes\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-config\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-var-lib-kubelet\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b010236-561c-4748-842a-648d3290ff2b-agent-certs\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-cnibin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-conf-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766654 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766673 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-ovn\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766692 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-bin\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-hosts-file\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-socket-dir-parent\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-kubelet\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766792 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766809 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-systemd\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-run\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766850 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-multus-certs\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-device-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766925 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-log-socket\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.767796 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-script-lib\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-host\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767013 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eddf108a-bf5f-483b-ac39-1804ea6d8d76-host-slash\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-netns\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767075 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-env-overrides\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767089 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-cni-binary-copy\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767089 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-bin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767145 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-node-log\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-dbus\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767167 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-log-socket\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766924 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-run\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-systemd-units\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9766570-5a16-4a4a-aa4e-900bbbed459c-serviceca\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767222 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-registration-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-k8s-cni-cncf-io\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767347 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-k8s-cni-cncf-io\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767355 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-system-cni-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.768535 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767389 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-systemd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767395 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767434 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767493 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-etc-kubernetes\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767675 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9766570-5a16-4a4a-aa4e-900bbbed459c-serviceca\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767741 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-ovn\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767785 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-bin\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767837 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-hosts-file\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767853 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-script-lib\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767884 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-socket-dir-parent\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767926 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-kubelet\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767931 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-host\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767947 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767970 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eddf108a-bf5f-483b-ac39-1804ea6d8d76-host-slash\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.767971 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-multus-daemon-config\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768011 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-kubelet-config\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768017 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-netns\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.769342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-config\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768063 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-bin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768066 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-device-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768070 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-systemd\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768064 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-conf\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766214 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-lib-modules\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.766265 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysconfig\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768148 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-cnibin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768613 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-system-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768644 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-sys-fs\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.768751 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-conf-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.769229 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-var-lib-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.769310 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-hostroot\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.769800 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-cnibin\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.770003 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-dbus\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.770084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-var-lib-kubelet\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.770262 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.770160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-tuned\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.771146 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.770796 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-registration-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.773461 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.771530 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-tmp\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.774065 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.774041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovn-node-metrics-cert\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.774182 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.774085 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b010236-561c-4748-842a-648d3290ff2b-agent-certs\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.775376 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.775353 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eddf108a-bf5f-483b-ac39-1804ea6d8d76-iptables-alerter-script\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.794405 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szsn4\" (UniqueName: \"kubernetes.io/projected/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-kube-api-access-szsn4\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.794471 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cwz\" (UniqueName: \"kubernetes.io/projected/17d8104e-e85c-490a-855d-63af3e241f11-kube-api-access-q4cwz\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.794477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdfb\" (UniqueName: \"kubernetes.io/projected/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-kube-api-access-nvdfb\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.794555 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.794575 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.794592 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:16.794694 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:16.794659 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:17.294640325 +0000 UTC m=+3.113631220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:16.796212 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.796162 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bjx\" (UniqueName: \"kubernetes.io/projected/f9766570-5a16-4a4a-aa4e-900bbbed459c-kube-api-access-v7bjx\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.796685 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.796665 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxk9\" (UniqueName: \"kubernetes.io/projected/0693da1f-8341-413f-b174-2575807a5a85-kube-api-access-4lxk9\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.797379 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.797356 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkwg\" (UniqueName: \"kubernetes.io/projected/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-kube-api-access-5lkwg\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.797528 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.797386 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfq8\" (UniqueName: \"kubernetes.io/projected/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-kube-api-access-pxfq8\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:16.798293 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.798274 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vfz\" (UniqueName: \"kubernetes.io/projected/28230167-b221-4cb3-8601-fe8bd9605a73-kube-api-access-g9vfz\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.799710 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.799685 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxvws\" (UniqueName: \"kubernetes.io/projected/eddf108a-bf5f-483b-ac39-1804ea6d8d76-kube-api-access-sxvws\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.890577 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.890540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:16.898078 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.898056 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:31:16.903385 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.903356 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b010236_561c_4748_842a_648d3290ff2b.slice/crio-ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6 WatchSource:0}: Error finding container ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6: Status 404 returned error can't find the container with id ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6 Oct 30 05:31:16.907943 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.907916 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8885b33d_691c_4ba4_916d_c1e27fbd1cd3.slice/crio-2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790 WatchSource:0}: Error finding container 2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790: Status 404 returned error can't find the container with id 2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790 Oct 30 05:31:16.912097 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.912075 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:31:16.923266 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.923235 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d8104e_e85c_490a_855d_63af3e241f11.slice/crio-825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247 WatchSource:0}: Error finding container 825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247: Status 404 returned error can't find the container with id 825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247 Oct 30 05:31:16.930477 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.930455 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:31:16.932880 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.932859 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:31:16.934488 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.934470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:16.938063 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.938038 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kcw92" Oct 30 05:31:16.941922 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.941835 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddf108a_bf5f_483b_ac39_1804ea6d8d76.slice/crio-8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78 WatchSource:0}: Error finding container 8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78: Status 404 returned error can't find the container with id 8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78 Oct 30 05:31:16.946170 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.946058 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9766570_5a16_4a4a_aa4e_900bbbed459c.slice/crio-100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997 WatchSource:0}: Error finding container 100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997: Status 404 returned error can't find the container with id 100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997 Oct 30 05:31:16.951538 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.951412 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28230167_b221_4cb3_8601_fe8bd9605a73.slice/crio-cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5 WatchSource:0}: Error finding container cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5: Status 404 returned error can't find the container with id cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5 Oct 30 05:31:16.953434 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.953381 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c85b4d_dc3f_4677_9fa8_d68410ea9b0b.slice/crio-319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508 WatchSource:0}: Error finding container 319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508: Status 404 returned error can't find the container with id 319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508 Oct 30 05:31:16.959546 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.959521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:31:16.961324 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:16.961302 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:31:16.970509 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.970483 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77f59d5_e2ae_46a9_b3b5_6333ec8ca595.slice/crio-0f9e70f36807393e9e22db8dfc312405291a27fd7b17e50ea49d1b93da8c7fb5 WatchSource:0}: Error finding container 0f9e70f36807393e9e22db8dfc312405291a27fd7b17e50ea49d1b93da8c7fb5: Status 404 returned error can't find the container with id 0f9e70f36807393e9e22db8dfc312405291a27fd7b17e50ea49d1b93da8c7fb5 Oct 30 05:31:16.976254 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:31:16.976217 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0693da1f_8341_413f_b174_2575807a5a85.slice/crio-1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1 WatchSource:0}: Error finding container 1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1: Status 404 returned error can't find the container with id 1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1 Oct 30 05:31:17.271494 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.271300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:17.271670 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.271547 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:17.271670 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.271460 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:17.271670 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.271629 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:17.271670 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.271637 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:31:18.271620367 +0000 UTC m=+4.090611243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:17.271805 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.271676 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:18.271663846 +0000 UTC m=+4.090654722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:17.372434 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.372355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:17.383760 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.381299 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:17.383760 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.381336 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:17.383760 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.381350 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:17.383760 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.381448 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:18.381420203 +0000 UTC m=+4.200411095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:17.611227 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.611156 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:26:16 +0000 UTC" deadline="2028-05-28 17:32:45.605814031 +0000 UTC" Oct 30 05:31:17.611227 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.611226 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13836h1m27.99459321s" Oct 30 05:31:17.684871 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.684788 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:17.685025 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:17.684931 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:17.696140 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.696033 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerStarted","Data":"1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1"} Oct 30 05:31:17.700203 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.700129 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kcw92" event={"ID":"28230167-b221-4cb3-8601-fe8bd9605a73","Type":"ContainerStarted","Data":"cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5"} Oct 30 05:31:17.708705 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.708644 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8mh7z" event={"ID":"f9766570-5a16-4a4a-aa4e-900bbbed459c","Type":"ContainerStarted","Data":"100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997"} Oct 30 05:31:17.714904 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.714866 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lp49f" event={"ID":"eddf108a-bf5f-483b-ac39-1804ea6d8d76","Type":"ContainerStarted","Data":"8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78"} Oct 30 05:31:17.720088 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.719993 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" event={"ID":"17d8104e-e85c-490a-855d-63af3e241f11","Type":"ContainerStarted","Data":"825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247"} Oct 30 05:31:17.724394 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.724365 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" event={"ID":"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595","Type":"ContainerStarted","Data":"0f9e70f36807393e9e22db8dfc312405291a27fd7b17e50ea49d1b93da8c7fb5"} Oct 30 05:31:17.743509 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.743475 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508"} Oct 30 05:31:17.748964 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.748873 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q2qpx" event={"ID":"8885b33d-691c-4ba4-916d-c1e27fbd1cd3","Type":"ContainerStarted","Data":"2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790"} Oct 30 05:31:17.753756 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.753631 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxrx6" event={"ID":"7b010236-561c-4748-842a-648d3290ff2b","Type":"ContainerStarted","Data":"ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6"} Oct 30 05:31:17.764160 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.763540 2583 generic.go:356] "Generic (PLEG): container finished" podID="a5d26ba6ff099574f4393ea9d38dc6f8" containerID="dc88da18c05c62df39a44ce17328550c8eca04b4c7c2629067863f46da5a840b" exitCode=0 Oct 30 05:31:17.764160 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.763620 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" event={"ID":"a5d26ba6ff099574f4393ea9d38dc6f8","Type":"ContainerDied","Data":"dc88da18c05c62df39a44ce17328550c8eca04b4c7c2629067863f46da5a840b"} Oct 30 05:31:17.798690 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.798208 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" event={"ID":"eddf44eee240eccf9696b703c5918959","Type":"ContainerStarted","Data":"da33e430f75db92257e6c4558cf2a07e3d58bf28c975663dcd2207e1b5b44e2c"} Oct 30 05:31:17.850422 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:17.850354 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" podStartSLOduration=2.850334451 podStartE2EDuration="2.850334451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:17.849917382 +0000 UTC m=+3.668908315" watchObservedRunningTime="2026-10-30 05:31:17.850334451 +0000 UTC m=+3.669325351" Oct 30 05:31:18.280521 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.280487 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:18.280699 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.280540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:18.280699 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.280668 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:18.280826 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.280723 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:31:20.280705118 +0000 UTC m=+6.099696009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:18.281157 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.281137 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:18.281249 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.281192 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:20.281177797 +0000 UTC m=+6.100168689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:18.482024 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.481339 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:18.482024 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.481486 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:18.482024 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.481501 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:18.482024 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.481518 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:18.482024 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.481565 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:20.481551195 +0000 UTC m=+6.300542088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:18.611586 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.611540 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:26:16 +0000 UTC" deadline="2028-07-27 23:55:50.828613882 +0000 UTC" Oct 30 05:31:18.611586 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.611584 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15282h24m32.217034758s" Oct 30 05:31:18.687226 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.687171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:18.687392 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.687314 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:18.687392 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.687348 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:18.687496 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:18.687448 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:18.819279 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.818686 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" event={"ID":"a5d26ba6ff099574f4393ea9d38dc6f8","Type":"ContainerStarted","Data":"db8b4253cf134b457d5c0fadd89fe1f0c40a9dab653ecf08942268f9952088b0"} Oct 30 05:31:18.861846 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:18.861792 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" podStartSLOduration=3.8617728209999997 podStartE2EDuration="3.861772821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:18.861383294 +0000 UTC m=+4.680374193" watchObservedRunningTime="2026-10-30 05:31:18.861772821 +0000 UTC m=+4.680763718" Oct 30 05:31:19.684635 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:19.684586 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:19.685140 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:19.684722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:20.300737 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.298172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:20.300737 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.298232 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:20.300737 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.298380 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:20.300737 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.298444 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:31:24.298427002 +0000 UTC m=+10.117417881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:20.300737 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.298855 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:20.300737 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.298896 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:24.298885521 +0000 UTC m=+10.117876396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:20.500899 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.500866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:20.501068 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.501030 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:20.501068 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.501048 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:20.501212 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.501059 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:20.501258 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.501217 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:24.501196887 +0000 UTC m=+10.320187777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:20.687533 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.687497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:20.687984 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.687644 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:20.688062 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.688023 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:20.688169 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:20.688148 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:20.836719 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.835887 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxrx6" event={"ID":"7b010236-561c-4748-842a-648d3290ff2b","Type":"ContainerStarted","Data":"8b970d0c4ebfb11f45a1724dad3432850c4de26f8be5d0d3914e20491dc4657a"} Oct 30 05:31:20.839354 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.838689 2583 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="a0b1ca3f72ea87186bf51607fcd5a51d38cf07d91706b3c725cc73fb1224f74f" exitCode=0 Oct 30 05:31:20.839354 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.838733 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerDied","Data":"a0b1ca3f72ea87186bf51607fcd5a51d38cf07d91706b3c725cc73fb1224f74f"} Oct 30 05:31:20.880079 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:20.879161 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sxrx6" podStartSLOduration=3.184905798 podStartE2EDuration="6.879142029s" totalImagesPullingTime="3.694236231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:20.878923128 +0000 UTC m=+6.697914024" watchObservedRunningTime="2026-10-30 05:31:20.879142029 +0000 UTC m=+6.698132932" Oct 30 05:31:21.685062 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:21.685033 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:21.685231 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:21.685198 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:21.847087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:21.846768 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8mh7z" event={"ID":"f9766570-5a16-4a4a-aa4e-900bbbed459c","Type":"ContainerStarted","Data":"1b6ccb9c5c020a336530e2f57916760b59ae9f5fc9df6821681be8eb0c8a480e"} Oct 30 05:31:21.889807 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:21.889747 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8mh7z" podStartSLOduration=3.317194367 podStartE2EDuration="7.889727141s" totalImagesPullingTime="4.572532774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:21.888947689 +0000 UTC m=+7.707938584" watchObservedRunningTime="2026-10-30 05:31:21.889727141 +0000 UTC m=+7.708718041" Oct 30 05:31:22.685263 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:22.685229 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:22.685445 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:22.685383 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:22.687748 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:22.687711 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:22.687882 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:22.687832 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:22.851263 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:22.851179 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" event={"ID":"17d8104e-e85c-490a-855d-63af3e241f11","Type":"ContainerStarted","Data":"3524d2f2a25717f4790ef51265dc3d0a91a88cb30fce37cbb0f0ae398f6b0737"} Oct 30 05:31:23.685139 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:23.685090 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:23.685327 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:23.685216 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:24.132521 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.132489 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:31:24.331976 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.331939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:24.332160 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.332003 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:24.332247 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.332160 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:24.332247 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.332210 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:24.332247 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.332223 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:31:32.332204913 +0000 UTC m=+18.151195811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:24.332247 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.332248 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:32.332237858 +0000 UTC m=+18.151228747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:24.533680 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.533568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:24.533847 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.533733 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:24.533847 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.533753 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:24.533847 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.533766 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:24.533847 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.533826 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:32.533808521 +0000 UTC m=+18.352799397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:24.617365 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.617290 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:31:24.132514816Z","UUID":"16896945-865e-4c73-bd67-d360c05acb74","Handler":null,"Name":"","Endpoint":""} Oct 30 05:31:24.622277 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.622253 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:31:24.622378 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.622301 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:31:24.685691 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.685652 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:24.685848 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.685786 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:24.686306 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.686282 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:24.686428 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:24.686383 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:24.858682 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.858647 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kcw92" event={"ID":"28230167-b221-4cb3-8601-fe8bd9605a73","Type":"ContainerStarted","Data":"0c0aa277345a3051fefdad948e2c1d6ce9c1ca2f80948f2e877ea0af7ee8a7a7"} Oct 30 05:31:24.861798 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:24.861766 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" event={"ID":"17d8104e-e85c-490a-855d-63af3e241f11","Type":"ContainerStarted","Data":"f7069e3773554c99dc187fe05bcf99fda350d67834eb3713355a2380851a3d4a"} Oct 30 05:31:25.685038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.684789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:25.685468 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:25.685182 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:25.798576 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.798535 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:25.799369 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.799347 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:25.843652 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.843583 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kcw92" podStartSLOduration=4.525277019 podStartE2EDuration="11.843564644s" totalImagesPullingTime="7.318287625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:24.904018792 +0000 UTC m=+10.723009690" watchObservedRunningTime="2026-10-30 05:31:25.843564644 +0000 UTC m=+11.662555546" Oct 30 05:31:25.867367 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.867322 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" event={"ID":"17d8104e-e85c-490a-855d-63af3e241f11","Type":"ContainerStarted","Data":"cdaa00168d88e0e006636749741b4d07d016f4b53c09569ad97372aee99211df"} Oct 30 05:31:25.870319 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.870247 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q2qpx" event={"ID":"8885b33d-691c-4ba4-916d-c1e27fbd1cd3","Type":"ContainerStarted","Data":"d2944abcb33db4ee336ef50b917b9a77d6911b871dc85b491aea6eaff9dc4ab6"} Oct 30 05:31:25.872989 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.872283 2583 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="a7d6ac2cacfb08c5cde909ae29d7540655ba74c05119727dc7bb3cf63e3412e8" exitCode=0 Oct 30 05:31:25.872989 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.872310 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerDied","Data":"a7d6ac2cacfb08c5cde909ae29d7540655ba74c05119727dc7bb3cf63e3412e8"} Oct 30 05:31:25.913226 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:25.913169 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" podStartSLOduration=4.065015428 podStartE2EDuration="11.913152077s" totalImagesPullingTime="7.848136649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:25.912843801 +0000 UTC m=+11.731834715" watchObservedRunningTime="2026-10-30 05:31:25.913152077 +0000 UTC m=+11.732142967" Oct 30 05:31:26.003762 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:26.003659 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q2qpx" podStartSLOduration=4.037455914 podStartE2EDuration="12.00363921s" totalImagesPullingTime="7.966183296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:26.003052605 +0000 UTC m=+11.822043503" watchObservedRunningTime="2026-10-30 05:31:26.00363921 +0000 UTC m=+11.822630109" Oct 30 05:31:26.684287 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:26.684250 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:26.684287 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:26.684298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:26.684575 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:26.684396 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:26.684575 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:26.684517 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:26.876286 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:26.876250 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lp49f" event={"ID":"eddf108a-bf5f-483b-ac39-1804ea6d8d76","Type":"ContainerStarted","Data":"04ee1acad2eefe1fb9d8a688dea7b6286ba220cca9ce7e74600e4e4a8f018b77"} Oct 30 05:31:26.921413 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:26.921353 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lp49f" podStartSLOduration=5.258462862 podStartE2EDuration="12.921333569s" totalImagesPullingTime="7.662870707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:26.921010208 +0000 UTC m=+12.740001106" watchObservedRunningTime="2026-10-30 05:31:26.921333569 +0000 UTC m=+12.740324469" Oct 30 05:31:27.685083 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:27.685042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:27.685265 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:27.685195 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:27.881178 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:27.881140 2583 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="0871a1c530bda2255403a1af15ee306d7a3055d3e891bc865d5bf881fe41ae84" exitCode=0 Oct 30 05:31:27.882011 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:27.881224 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerDied","Data":"0871a1c530bda2255403a1af15ee306d7a3055d3e891bc865d5bf881fe41ae84"} Oct 30 05:31:27.882930 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:27.882899 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" event={"ID":"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595","Type":"ContainerStarted","Data":"7182bee4147602c428dd8381d9a0ba7b20fcddebd381701cc148bfbc2d5b01d7"} Oct 30 05:31:27.993304 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:27.993206 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" podStartSLOduration=3.885953128 podStartE2EDuration="13.993192167s" totalImagesPullingTime="10.107239039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:27.993095196 +0000 UTC m=+13.812086095" watchObservedRunningTime="2026-10-30 05:31:27.993192167 +0000 UTC m=+13.812183066" Oct 30 05:31:28.684728 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:28.684688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:28.684854 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:28.684728 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:28.684905 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:28.684844 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:28.685041 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:28.684990 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:28.888092 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:28.888057 2583 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="85b877f536d3283901c73ca98a6675ccb80e0cdd82b68728fe4cad2aec925adf" exitCode=0 Oct 30 05:31:28.888569 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:28.888151 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerDied","Data":"85b877f536d3283901c73ca98a6675ccb80e0cdd82b68728fe4cad2aec925adf"} Oct 30 05:31:29.684448 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:29.684191 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:29.684612 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:29.684540 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:30.684806 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:30.684772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:30.684806 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:30.684803 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:30.685349 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:30.684911 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:30.685349 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:30.685056 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:31.684716 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.684527 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:31.684894 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:31.684788 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:31.897506 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.897419 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"78d8d846dd593cd4bd6a365eddd605a8b94d7542b74ce250cd34fdf45a88c0e2"} Oct 30 05:31:31.897506 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.897495 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"eeaf669508ff10a8559cf7cd6f991b383aaab3bfc24b3225c3495250beefd849"} Oct 30 05:31:31.897676 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.897525 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"f6ebe9a8dd66ce358af54e14444b382a3191d640bc7058593517b4af2a9394ad"} Oct 30 05:31:31.897676 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.897564 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"feb6109dc199657c082670e0c5a26ebc84a876aaf94bf9477be23f0d8b4cb45a"} Oct 30 05:31:31.897676 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.897591 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"c148a5d60797b7f19ec65f00fa5a0091c8f2647628d2f9490b3eeb69acf56b3d"} Oct 30 05:31:31.897676 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.897624 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"c6f6bbcf8079cc419071f2cb6bf8976d626bf887e519975200d0f013410d8694"} Oct 30 05:31:31.900186 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.900145 2583 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="736b381e82ba0ddb679aca1bd2645a9c7e268bc84b8506b347d3749eaf2e1ba2" exitCode=0 Oct 30 05:31:31.900186 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:31.900192 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerDied","Data":"736b381e82ba0ddb679aca1bd2645a9c7e268bc84b8506b347d3749eaf2e1ba2"} Oct 30 05:31:32.394391 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.394297 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:32.394391 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.394362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:32.394575 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.394453 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:32.394575 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.394457 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:32.394575 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.394502 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:48.394491055 +0000 UTC m=+34.213481931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:32.394575 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.394517 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:31:48.394508797 +0000 UTC m=+34.213499674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:32.595501 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.595464 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:32.595656 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.595631 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:32.595656 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.595652 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:32.595738 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.595661 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:32.595738 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.595714 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:48.595699311 +0000 UTC m=+34.414690188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:32.684593 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.684510 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:32.684738 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.684634 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:32.684738 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.684694 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:32.684838 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:32.684818 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:32.905174 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.905140 2583 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="570486ff266db57732feed3ff8350508da23a3fa08e69c43c17576b4ebeb965e" exitCode=0 Oct 30 05:31:32.905637 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:32.905201 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerDied","Data":"570486ff266db57732feed3ff8350508da23a3fa08e69c43c17576b4ebeb965e"} Oct 30 05:31:33.684224 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:33.684187 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:33.684569 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:33.684315 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:33.910527 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:33.910438 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" event={"ID":"0693da1f-8341-413f-b174-2575807a5a85","Type":"ContainerStarted","Data":"72bf019a54cc43c99bd86a3eca3d385c108edf16c07de7be09419bdab9f905e2"} Oct 30 05:31:33.913713 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:33.913690 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"1f5d5940de8e181b3f366be2b6ac1096836f017d2df88aa3f8a02dcef19feece"} Oct 30 05:31:33.988160 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:33.988084 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pcx4f" podStartSLOduration=9.393697361 podStartE2EDuration="19.988071617s" totalImagesPullingTime="10.594374256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:33.987582771 +0000 UTC m=+19.806573670" watchObservedRunningTime="2026-10-30 05:31:33.988071617 +0000 UTC m=+19.807062514" Oct 30 05:31:34.242728 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:34.242658 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2qpx_8885b33d-691c-4ba4-916d-c1e27fbd1cd3/dns-node-resolver/0.log" Oct 30 05:31:34.685608 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:34.685531 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:34.685742 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:34.685612 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:34.685742 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:34.685705 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:34.685811 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:34.685791 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:34.779570 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:34.779543 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mh7z_f9766570-5a16-4a4a-aa4e-900bbbed459c/node-ca/0.log" Oct 30 05:31:35.684187 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:35.684160 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:35.684871 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:35.684257 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:35.922263 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:35.922177 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" event={"ID":"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b","Type":"ContainerStarted","Data":"27d2849c15ee1009582bfbafcb49f12a07cfbc772bc0df2ab1a99e7637721bd3"} Oct 30 05:31:35.922654 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:35.922628 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:35.922775 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:35.922687 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:35.939045 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:35.939018 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:36.097215 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:36.096993 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" podStartSLOduration=7.820964652 podStartE2EDuration="22.096979498s" totalImagesPullingTime="14.276014846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:36.025786821 +0000 UTC m=+21.844777729" watchObservedRunningTime="2026-10-30 05:31:36.096979498 +0000 UTC m=+21.915970396" Oct 30 05:31:36.684507 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:36.684470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:36.684507 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:36.684496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:36.685016 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:36.684593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:36.685016 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:36.684695 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:36.924765 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:36.924732 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:36.940709 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:36.940631 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:31:37.684192 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:37.684159 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:37.684353 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:37.684265 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:38.684791 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:38.684752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:38.684791 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:38.684777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:38.685218 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:38.684862 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:38.685218 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:38.684981 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:39.043632 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:39.043558 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:39.043778 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:39.043668 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:31:39.044175 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:39.044155 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:31:39.684817 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:39.684781 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:39.685270 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:39.684886 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:40.684448 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:40.684413 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:40.684620 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:40.684544 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:40.684620 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:40.684591 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:40.684773 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:40.684670 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:41.684797 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:41.684767 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:41.685268 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:41.684863 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:42.684981 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:42.684947 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:42.685439 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:42.685051 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:42.685439 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:42.685180 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:42.685439 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:42.685278 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:43.684863 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:43.684812 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:43.685054 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:43.684933 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:44.684998 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:44.684958 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:44.685217 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:44.685170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:44.685217 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:44.685165 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:44.685645 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:44.685271 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:45.684303 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:45.684269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:45.684503 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:45.684382 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:46.684644 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:46.684603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:46.685073 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:46.684657 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:46.685073 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:46.684782 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:46.685073 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:46.684891 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:47.684717 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:47.684674 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:47.685169 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:47.684801 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:48.404976 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:48.404934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:48.404976 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:48.404986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:48.405241 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.405084 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:48.405241 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.405085 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:48.405241 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.405154 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret podName:20949c65-577f-4bef-9494-b8ddec17c982 nodeName:}" failed. No retries permitted until 2026-10-30 05:32:20.405139825 +0000 UTC m=+66.224130719 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret") pod "global-pull-secret-syncer-wb5rz" (UID: "20949c65-577f-4bef-9494-b8ddec17c982") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:48.405241 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.405168 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs podName:16cdbe91-7a39-4ad3-8ef5-c738a995e26a nodeName:}" failed. No retries permitted until 2026-10-30 05:32:20.405161821 +0000 UTC m=+66.224152696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs") pod "network-metrics-daemon-4rx42" (UID: "16cdbe91-7a39-4ad3-8ef5-c738a995e26a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:48.606913 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:48.606869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:48.607085 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.607026 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:48.607085 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.607042 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:48.607085 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.607051 2583 projected.go:196] Error preparing data for projected volume kube-api-access-8ssgs for pod openshift-network-diagnostics/network-check-target-phtr7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:48.607210 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.607120 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs podName:794c8751-becf-4dc4-8574-282defa3eb69 nodeName:}" failed. No retries permitted until 2026-10-30 05:32:20.607087656 +0000 UTC m=+66.426078533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8ssgs" (UniqueName: "kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs") pod "network-check-target-phtr7" (UID: "794c8751-becf-4dc4-8574-282defa3eb69") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:48.685277 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:48.685188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:48.685738 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:48.685188 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:48.685738 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.685334 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:48.685738 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:48.685412 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:49.684629 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:49.684575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:49.684816 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:49.684697 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:50.684915 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:50.684863 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:50.685444 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:50.684987 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:50.685444 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:50.685062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:50.685444 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:50.685187 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:51.684603 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:51.684569 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:51.684793 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:51.684666 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:52.684382 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:52.684341 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:52.684908 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:52.684489 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:52.684908 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:52.684545 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:52.684908 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:52.684650 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:53.685035 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:53.684996 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:53.685455 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:53.685102 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:54.690793 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.690756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:54.691342 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:54.690910 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:54.691342 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.690991 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:54.691342 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:54.691148 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:54.805199 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.805119 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wb5rz"] Oct 30 05:31:54.808793 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.808766 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-phtr7"] Oct 30 05:31:54.808911 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.808818 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rx42"] Oct 30 05:31:54.808911 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.808890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:54.808981 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:54.808959 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:54.970323 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.970294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:54.970488 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:54.970294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:54.970488 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:54.970410 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:54.970488 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:54.970468 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:55.520496 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:55.520468 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2qpx_8885b33d-691c-4ba4-916d-c1e27fbd1cd3/dns-node-resolver/0.log" Oct 30 05:31:56.321253 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:56.321069 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mh7z_f9766570-5a16-4a4a-aa4e-900bbbed459c/node-ca/0.log" Oct 30 05:31:56.684734 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:56.684630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:56.684734 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:56.684651 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:56.684961 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:56.684773 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:56.684961 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:56.684786 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:56.684961 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:56.684867 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:56.684961 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:56.684920 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:31:58.688038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:58.688004 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:31:58.688038 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:58.688036 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:31:58.688616 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:31:58.688171 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:31:58.688616 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:58.688185 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4rx42" podUID="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" Oct 30 05:31:58.688616 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:58.688266 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wb5rz" podUID="20949c65-577f-4bef-9494-b8ddec17c982" Oct 30 05:31:58.688616 ip-10-0-138-15 kubenswrapper[2583]: E1030 05:31:58.688297 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-phtr7" podUID="794c8751-becf-4dc4-8574-282defa3eb69" Oct 30 05:32:00.005366 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.005337 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeReady" Oct 30 05:32:00.005806 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.005464 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:32:00.068997 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.068974 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rm8hh"] Oct 30 05:32:00.070806 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.070790 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.074670 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.074646 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b896w"] Oct 30 05:32:00.076739 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.076708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.089824 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.089800 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ctntz"] Oct 30 05:32:00.091863 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.091838 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.092524 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.092502 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvn8d\"" Oct 30 05:32:00.092648 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.092587 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:32:00.093178 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.093157 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:32:00.093411 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.093395 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rm8hh"] Oct 30 05:32:00.094373 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.094357 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:32:00.101925 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.101904 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b896w"] Oct 30 05:32:00.115362 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.115337 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ctntz"] Oct 30 05:32:00.181891 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.181853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5818964-b07b-4362-9eba-2458e3d58a97-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.181891 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.181894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5818964-b07b-4362-9eba-2458e3d58a97-data-volume\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.182099 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.181914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-tmp-dir\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.182099 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.181976 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-config-volume\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.182099 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-metrics-tls\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.182099 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182064 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txdt\" (UniqueName: \"kubernetes.io/projected/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-kube-api-access-2txdt\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.182099 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5818964-b07b-4362-9eba-2458e3d58a97-crio-socket\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.182321 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182145 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8g9\" (UniqueName: \"kubernetes.io/projected/d5818964-b07b-4362-9eba-2458e3d58a97-kube-api-access-qn8g9\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.182321 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbwb\" (UniqueName: \"kubernetes.io/projected/a9cd7570-1937-4334-90a6-8c4d620a2216-kube-api-access-mqbwb\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.182321 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182301 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5818964-b07b-4362-9eba-2458e3d58a97-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.182408 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.182328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cd7570-1937-4334-90a6-8c4d620a2216-cert\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.192767 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.192734 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:32:00.192914 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.192879 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:32:00.192992 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.192959 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:32:00.193049 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.192990 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bmrwn\"" Oct 30 05:32:00.193049 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.193003 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:32:00.193049 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.193036 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:32:00.193049 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.193047 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:32:00.193214 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.193067 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7pqs8\"" Oct 30 05:32:00.282783 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5818964-b07b-4362-9eba-2458e3d58a97-crio-socket\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.282783 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282783 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8g9\" (UniqueName: \"kubernetes.io/projected/d5818964-b07b-4362-9eba-2458e3d58a97-kube-api-access-qn8g9\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbwb\" (UniqueName: \"kubernetes.io/projected/a9cd7570-1937-4334-90a6-8c4d620a2216-kube-api-access-mqbwb\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5818964-b07b-4362-9eba-2458e3d58a97-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cd7570-1937-4334-90a6-8c4d620a2216-cert\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5818964-b07b-4362-9eba-2458e3d58a97-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5818964-b07b-4362-9eba-2458e3d58a97-data-volume\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-tmp-dir\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.282979 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-config-volume\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.283033 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5818964-b07b-4362-9eba-2458e3d58a97-crio-socket\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.283440 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283232 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-metrics-tls\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.283440 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283268 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txdt\" (UniqueName: \"kubernetes.io/projected/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-kube-api-access-2txdt\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.283440 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283399 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-tmp-dir\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.283563 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283466 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5818964-b07b-4362-9eba-2458e3d58a97-data-volume\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.283712 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283686 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-config-volume\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.283712 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.283707 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5818964-b07b-4362-9eba-2458e3d58a97-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.287369 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.287342 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5818964-b07b-4362-9eba-2458e3d58a97-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.287471 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.287345 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-metrics-tls\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.287471 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.287394 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cd7570-1937-4334-90a6-8c4d620a2216-cert\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.309285 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.309258 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8g9\" (UniqueName: \"kubernetes.io/projected/d5818964-b07b-4362-9eba-2458e3d58a97-kube-api-access-qn8g9\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.309492 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.309471 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txdt\" (UniqueName: \"kubernetes.io/projected/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-kube-api-access-2txdt\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.309558 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.309544 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbwb\" (UniqueName: \"kubernetes.io/projected/a9cd7570-1937-4334-90a6-8c4d620a2216-kube-api-access-mqbwb\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.383503 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.383408 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:00.388182 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.388164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:00.405157 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.405133 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:00.685147 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.685055 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:00.685147 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.685128 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:00.685354 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.685134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:00.693619 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.693592 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b896w"] Oct 30 05:32:00.697457 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:00.697430 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd7570_1937_4334_90a6_8c4d620a2216.slice/crio-aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6 WatchSource:0}: Error finding container aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6: Status 404 returned error can't find the container with id aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6 Oct 30 05:32:00.704922 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.704802 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:32:00.708600 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.708572 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:32:00.708721 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.708633 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:32:00.710176 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.710149 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rm8hh"] Oct 30 05:32:00.715599 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:00.715576 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334ad7cd_4277_4572_b7e2_b0e2ececd5ff.slice/crio-a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c WatchSource:0}: Error finding container a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c: Status 404 returned error can't find the container with id a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c Oct 30 05:32:00.723473 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.723451 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ctntz"] Oct 30 05:32:00.726019 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:00.725997 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5818964_b07b_4362_9eba_2458e3d58a97.slice/crio-c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265 WatchSource:0}: Error finding container c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265: Status 404 returned error can't find the container with id c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265 Oct 30 05:32:00.805372 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.805337 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:32:00.805372 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.805364 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8jj4\"" Oct 30 05:32:00.805520 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.805364 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nr89r\"" Oct 30 05:32:00.805520 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.805413 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wks87\"" Oct 30 05:32:00.984157 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.984052 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rm8hh" event={"ID":"334ad7cd-4277-4572-b7e2-b0e2ececd5ff","Type":"ContainerStarted","Data":"a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c"} Oct 30 05:32:00.985469 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.985446 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctntz" event={"ID":"d5818964-b07b-4362-9eba-2458e3d58a97","Type":"ContainerStarted","Data":"03b9d7efdfe750967adea90dadfa86e8de1328e9c42a5de3727656380f299902"} Oct 30 05:32:00.985588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.985500 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctntz" event={"ID":"d5818964-b07b-4362-9eba-2458e3d58a97","Type":"ContainerStarted","Data":"c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265"} Oct 30 05:32:00.986506 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:00.986481 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b896w" event={"ID":"a9cd7570-1937-4334-90a6-8c4d620a2216","Type":"ContainerStarted","Data":"aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6"} Oct 30 05:32:01.992644 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:01.992598 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctntz" event={"ID":"d5818964-b07b-4362-9eba-2458e3d58a97","Type":"ContainerStarted","Data":"7e4cf84fde3a8a3503045c13884b92d9fa64acddac4c9b5b20e7cd6518f72fe8"} Oct 30 05:32:02.313362 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.313241 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk"] Oct 30 05:32:02.315597 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.315572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.336730 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.336691 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk"] Oct 30 05:32:02.351068 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.349950 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wdgdh\"" Oct 30 05:32:02.351068 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.350775 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:32:02.351068 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.350875 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:32:02.351068 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.350926 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 05:32:02.351068 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.351013 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:32:02.353824 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.353798 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:32:02.365751 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.365717 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s"] Oct 30 05:32:02.367937 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.367913 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-68zhv"] Oct 30 05:32:02.368075 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.368057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.370187 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.370153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.373890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.373865 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s"] Oct 30 05:32:02.400912 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.400871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hdv\" (UniqueName: \"kubernetes.io/projected/98584416-9e1b-4358-8a58-23d4b123be72-kube-api-access-q4hdv\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.401084 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.400940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.401084 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.400977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98584416-9e1b-4358-8a58-23d4b123be72-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.401275 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401227 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.401469 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401445 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:32:02.401564 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401509 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x96j8\"" Oct 30 05:32:02.401615 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401589 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 05:32:02.401710 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401696 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:32:02.401977 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401857 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lg7qg\"" Oct 30 05:32:02.401977 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.401940 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:32:02.439620 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.439590 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:32:02.439914 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.439895 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 05:32:02.501719 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f8b282a1-edcf-42d2-9f53-c995efc488cd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.501890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501738 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.501890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbvz\" (UniqueName: \"kubernetes.io/projected/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-api-access-9wbvz\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.501890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-tls\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.501890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501855 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.501890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501891 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-wtmp\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6c4\" (UniqueName: \"kubernetes.io/projected/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-kube-api-access-pl6c4\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.501993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-sys\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502019 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-metrics-client-ca\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hdv\" (UniqueName: \"kubernetes.io/projected/98584416-9e1b-4358-8a58-23d4b123be72-kube-api-access-q4hdv\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.502087 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502059 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.502359 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.502359 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-root\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502359 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-accelerators-collector-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502359 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502238 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-textfile\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.502359 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98584416-9e1b-4358-8a58-23d4b123be72-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.502760 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.502587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.503299 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.503159 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98584416-9e1b-4358-8a58-23d4b123be72-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.506009 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.505948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.506009 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.505992 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.528907 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.528871 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hdv\" (UniqueName: \"kubernetes.io/projected/98584416-9e1b-4358-8a58-23d4b123be72-kube-api-access-q4hdv\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbvz\" (UniqueName: \"kubernetes.io/projected/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-api-access-9wbvz\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-tls\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603326 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-wtmp\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603368 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603407 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6c4\" (UniqueName: \"kubernetes.io/projected/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-kube-api-access-pl6c4\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-sys\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-metrics-client-ca\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603467 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603483 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-root\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-accelerators-collector-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-textfile\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603560 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f8b282a1-edcf-42d2-9f53-c995efc488cd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.604901 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.603585 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.606588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.605460 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-root\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.606588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.605506 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-sys\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.606588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.605538 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-wtmp\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.606588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.605750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-textfile\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.606588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.605887 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f8b282a1-edcf-42d2-9f53-c995efc488cd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.606588 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.606304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.607035 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.606607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.607035 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.606683 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-metrics-client-ca\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.607248 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.607180 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-accelerators-collector-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.610329 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.609994 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.610329 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.610132 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.610329 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.610152 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-tls\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.611130 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.611090 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.630128 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.630078 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbvz\" (UniqueName: \"kubernetes.io/projected/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-api-access-9wbvz\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.630503 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.630469 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6c4\" (UniqueName: \"kubernetes.io/projected/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-kube-api-access-pl6c4\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.631714 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.631688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:02.691385 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.691360 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:02.694724 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.693734 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:02.711670 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:02.711641 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714aff30_d1b2_4ed7_9a37_e7674c4c2ad3.slice/crio-0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb WatchSource:0}: Error finding container 0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb: Status 404 returned error can't find the container with id 0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb Oct 30 05:32:02.966234 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.966200 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk"] Oct 30 05:32:02.970931 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:02.970900 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98584416_9e1b_4358_8a58_23d4b123be72.slice/crio-48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171 WatchSource:0}: Error finding container 48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171: Status 404 returned error can't find the container with id 48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171 Oct 30 05:32:02.998704 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.998672 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rm8hh" event={"ID":"334ad7cd-4277-4572-b7e2-b0e2ececd5ff","Type":"ContainerStarted","Data":"5edf87bef6bdb20d09b4a0f647c2bd999bff0aa9c94152e0579026714864dcd4"} Oct 30 05:32:02.999010 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.998742 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rm8hh" event={"ID":"334ad7cd-4277-4572-b7e2-b0e2ececd5ff","Type":"ContainerStarted","Data":"a467a334c4d749de3abfb2c129d8a990a42941dec2409c648bc569bd84c414ba"} Oct 30 05:32:02.999484 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:02.999467 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:03.000761 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:03.000739 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" event={"ID":"98584416-9e1b-4358-8a58-23d4b123be72","Type":"ContainerStarted","Data":"48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171"} Oct 30 05:32:03.002009 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:03.001986 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68zhv" event={"ID":"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3","Type":"ContainerStarted","Data":"0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb"} Oct 30 05:32:03.013116 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:03.013079 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s"] Oct 30 05:32:03.015959 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:03.015778 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b282a1_edcf_42d2_9f53_c995efc488cd.slice/crio-15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875 WatchSource:0}: Error finding container 15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875: Status 404 returned error can't find the container with id 15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875 Oct 30 05:32:03.049013 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:03.048895 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rm8hh" podStartSLOduration=1.07604973 podStartE2EDuration="3.048881359s" totalImagesPullingTime="1.972831629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:03.048631701 +0000 UTC m=+48.867622606" watchObservedRunningTime="2026-10-30 05:32:03.048881359 +0000 UTC m=+48.867872257" Oct 30 05:32:04.012423 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.012340 2583 generic.go:356] "Generic (PLEG): container finished" podID="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" containerID="9d3a59d1c3bd2bf91d2b308d93b0f8518d0bd7a2d8c0f955cea8b57d6dd2c850" exitCode=0 Oct 30 05:32:04.012809 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.012435 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68zhv" event={"ID":"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3","Type":"ContainerDied","Data":"9d3a59d1c3bd2bf91d2b308d93b0f8518d0bd7a2d8c0f955cea8b57d6dd2c850"} Oct 30 05:32:04.014906 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.014871 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ctntz" event={"ID":"d5818964-b07b-4362-9eba-2458e3d58a97","Type":"ContainerStarted","Data":"abe7d9fd948541bdce6d00665974e706948409070a6500a7d6e41032ba61e251"} Oct 30 05:32:04.017147 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.017096 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" event={"ID":"98584416-9e1b-4358-8a58-23d4b123be72","Type":"ContainerStarted","Data":"38830cb3c964ccb2bf86ffd3a7b31b62b42418397b32724420348532af87ad70"} Oct 30 05:32:04.017255 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.017172 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" event={"ID":"98584416-9e1b-4358-8a58-23d4b123be72","Type":"ContainerStarted","Data":"ce127356b7de33654121b5511aa224405edd6ec0157fa558b10da55db4981cab"} Oct 30 05:32:04.019053 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.019030 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b896w" event={"ID":"a9cd7570-1937-4334-90a6-8c4d620a2216","Type":"ContainerStarted","Data":"b847e56736a0879fc84b3b1467716a4570e870cc5b3e011eca319fceb295aa8d"} Oct 30 05:32:04.020271 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.020251 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" event={"ID":"f8b282a1-edcf-42d2-9f53-c995efc488cd","Type":"ContainerStarted","Data":"15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875"} Oct 30 05:32:04.108414 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.108352 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ctntz" podStartSLOduration=1.8251977560000001 podStartE2EDuration="4.108331258s" totalImagesPullingTime="2.283133502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:32:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:04.106847399 +0000 UTC m=+49.925838335" watchObservedRunningTime="2026-10-30 05:32:04.108331258 +0000 UTC m=+49.927322159" Oct 30 05:32:04.150604 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:04.150549 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b896w" podStartSLOduration=1.040990047 podStartE2EDuration="4.150535377s" totalImagesPullingTime="3.10954533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:04.149773001 +0000 UTC m=+49.968763928" watchObservedRunningTime="2026-10-30 05:32:04.150535377 +0000 UTC m=+49.969526276" Oct 30 05:32:05.025660 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.025623 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" event={"ID":"98584416-9e1b-4358-8a58-23d4b123be72","Type":"ContainerStarted","Data":"9b8bc5d60b96b2d22b536bbade1f02ee1444cdf74fb4aad6c47a4693f5dbc097"} Oct 30 05:32:05.027665 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.027632 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" event={"ID":"f8b282a1-edcf-42d2-9f53-c995efc488cd","Type":"ContainerStarted","Data":"b146ff7fcb24f0d2c74dc4a618671be9387dafde4f2236204f0110b7f8f459b8"} Oct 30 05:32:05.027800 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.027692 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" event={"ID":"f8b282a1-edcf-42d2-9f53-c995efc488cd","Type":"ContainerStarted","Data":"9ba8b19c4045ddbd45db918d528e1dcb776f43f6dc58922a1be3d23f9b810f1b"} Oct 30 05:32:05.027800 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.027711 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" event={"ID":"f8b282a1-edcf-42d2-9f53-c995efc488cd","Type":"ContainerStarted","Data":"4f0d2f9c3559420e8e254f2c8c7bccab88c10123e133547b6ed19afad3b7f0f9"} Oct 30 05:32:05.029701 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.029677 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68zhv" event={"ID":"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3","Type":"ContainerStarted","Data":"184eefecb3eac2376d5f99f33756f52645cccd1f4547e1d46d2324e6eea5f4d3"} Oct 30 05:32:05.029768 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.029720 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-68zhv" event={"ID":"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3","Type":"ContainerStarted","Data":"2690ef7bedfdcc30d94e43051bba0289811294de9c584210a704ced6dd5964d3"} Oct 30 05:32:05.072453 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.072407 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" podStartSLOduration=1.910727627 podStartE2EDuration="3.072396246s" totalImagesPullingTime="1.161668619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:05.071941393 +0000 UTC m=+50.890932335" watchObservedRunningTime="2026-10-30 05:32:05.072396246 +0000 UTC m=+50.891387143" Oct 30 05:32:05.121595 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.121548 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-68zhv" podStartSLOduration=2.060988775 podStartE2EDuration="3.121535059s" totalImagesPullingTime="963.779858ms" totalInitContainerRuntime="96.766426ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:05.12060154 +0000 UTC m=+50.939592438" watchObservedRunningTime="2026-10-30 05:32:05.121535059 +0000 UTC m=+50.940525981" Oct 30 05:32:05.169493 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:05.169437 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" podStartSLOduration=1.8203765600000001 podStartE2EDuration="3.169425248s" totalImagesPullingTime="1.349048688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:05.168766456 +0000 UTC m=+50.987757356" watchObservedRunningTime="2026-10-30 05:32:05.169425248 +0000 UTC m=+50.988416157" Oct 30 05:32:07.138643 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.138605 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq"] Oct 30 05:32:07.141006 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.140984 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.159400 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.159368 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 05:32:07.160328 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.160307 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq"] Oct 30 05:32:07.173463 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.173437 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t7gdv\"" Oct 30 05:32:07.248208 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.248168 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.248385 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.248281 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29gt\" (UniqueName: \"kubernetes.io/projected/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-kube-api-access-n29gt\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.348925 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.348886 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29gt\" (UniqueName: \"kubernetes.io/projected/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-kube-api-access-n29gt\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.349075 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.348955 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.351567 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.351541 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.374554 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.374516 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29gt\" (UniqueName: \"kubernetes.io/projected/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-kube-api-access-n29gt\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.451043 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.450962 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:07.745237 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:07.745163 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq"] Oct 30 05:32:07.748139 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:07.748090 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff6f637_e30a_47b8_88fe_a5aa3dfd5e02.slice/crio-6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89 WatchSource:0}: Error finding container 6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89: Status 404 returned error can't find the container with id 6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89 Oct 30 05:32:08.039750 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:08.039674 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" event={"ID":"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02","Type":"ContainerStarted","Data":"6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89"} Oct 30 05:32:08.941228 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:08.941197 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:09.328142 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.328090 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:09.330711 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.330688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.349870 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.349847 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:32:09.350014 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.349998 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:32:09.350092 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.350078 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:32:09.350487 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.350467 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:32:09.365327 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365296 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:32:09.365477 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365331 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:32:09.365477 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365381 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:32:09.365477 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365393 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:32:09.365477 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365471 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365501 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mkgv\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365566 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365568 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2eaov2pk6s7tl\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365615 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365620 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365682 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:32:09.365703 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.365722 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:32:09.367151 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.367136 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:09.379276 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.379257 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:32:09.469904 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.469866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.469904 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.469910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470132 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.469934 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470132 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470132 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470072 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470132 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470283 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470283 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470380 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470300 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470380 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470439 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470405 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470490 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470490 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470466 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470578 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470512 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470578 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470548 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470664 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470580 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470664 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470740 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470678 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470740 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470694 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.470740 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.470711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571484 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571484 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571720 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571505 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571720 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571720 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571720 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571634 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571720 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571694 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571722 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571750 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.571873 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571857 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572055 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571888 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572055 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571903 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572055 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572055 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572055 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572055 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.571980 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.572316 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.572288 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.574883 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.572770 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.574883 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.573065 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.574883 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.573691 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.574883 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.574748 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.575791 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.575374 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.575791 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.575387 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.575791 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.575554 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.575989 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.575806 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.575989 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.575834 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.575989 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.575926 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.576161 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.576068 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.576566 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.576510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.576753 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.576729 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.577541 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.577518 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.577755 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.577738 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.578136 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.578093 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.578662 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.578615 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.578749 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.578721 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.600929 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.597654 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.647360 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.647325 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:09.988980 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:09.988951 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:09.991291 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:09.991264 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abeb78e_0d20_44fe_a201_5855719acd9b.slice/crio-858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a WatchSource:0}: Error finding container 858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a: Status 404 returned error can't find the container with id 858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a Oct 30 05:32:10.047674 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:10.047640 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a"} Oct 30 05:32:10.049219 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:10.049191 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" event={"ID":"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02","Type":"ContainerStarted","Data":"cad08440be9727c456f4412bac3b38d575e77cbfbda744e021f92b444fb26596"} Oct 30 05:32:10.049399 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:10.049380 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:10.054447 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:10.054429 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:10.095199 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:10.095153 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" podStartSLOduration=1.826106472 podStartE2EDuration="3.095139505s" totalImagesPullingTime="1.269033033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:10.094824208 +0000 UTC m=+55.913815107" watchObservedRunningTime="2026-10-30 05:32:10.095139505 +0000 UTC m=+55.914130403" Oct 30 05:32:11.054340 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:11.054301 2583 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" exitCode=0 Oct 30 05:32:11.054892 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:11.054389 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} Oct 30 05:32:14.038530 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:14.038509 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:15.072865 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:15.072807 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} Oct 30 05:32:15.073266 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:15.072910 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} Oct 30 05:32:16.079144 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:16.079097 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} Oct 30 05:32:17.085589 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:17.085552 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} Oct 30 05:32:17.085990 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:17.085623 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} Oct 30 05:32:17.085990 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:17.085656 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerStarted","Data":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} Oct 30 05:32:17.158881 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:17.158831 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.526570833 podStartE2EDuration="8.158817439s" totalImagesPullingTime="5.616156907s" totalInitContainerRuntime="16.089699ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:17.158224996 +0000 UTC m=+62.977215895" watchObservedRunningTime="2026-10-30 05:32:17.158817439 +0000 UTC m=+62.977808336" Oct 30 05:32:19.648426 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:19.648393 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:20.464963 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.464924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:20.465153 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.465005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:20.480807 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.480781 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:32:20.488517 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.488481 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:20.494828 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.494809 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:32:20.497910 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.497893 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:20.526685 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.526662 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8jj4\"" Oct 30 05:32:20.531725 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.531700 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:20.666136 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.666071 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:20.681866 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.681835 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:32:20.701844 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.701818 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:32:20.709610 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.709576 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:20.813575 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.813495 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wks87\"" Oct 30 05:32:20.817615 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.817593 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:20.819256 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.819233 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nr89r\"" Oct 30 05:32:20.824208 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.824191 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:20.831675 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:20.831653 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4rx42"] Oct 30 05:32:20.835268 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:20.835237 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cdbe91_7a39_4ad3_8ef5_c738a995e26a.slice/crio-48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05 WatchSource:0}: Error finding container 48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05: Status 404 returned error can't find the container with id 48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05 Oct 30 05:32:21.099415 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:21.099376 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rx42" event={"ID":"16cdbe91-7a39-4ad3-8ef5-c738a995e26a","Type":"ContainerStarted","Data":"48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05"} Oct 30 05:32:21.120863 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:21.120831 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wb5rz"] Oct 30 05:32:21.123584 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:21.123552 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20949c65_577f_4bef_9494_b8ddec17c982.slice/crio-2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de WatchSource:0}: Error finding container 2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de: Status 404 returned error can't find the container with id 2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de Oct 30 05:32:21.134446 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:21.134423 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-phtr7"] Oct 30 05:32:21.137344 ip-10-0-138-15 kubenswrapper[2583]: W1030 05:32:21.137319 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794c8751_becf_4dc4_8574_282defa3eb69.slice/crio-252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e WatchSource:0}: Error finding container 252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e: Status 404 returned error can't find the container with id 252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e Oct 30 05:32:22.105890 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:22.105854 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rx42" event={"ID":"16cdbe91-7a39-4ad3-8ef5-c738a995e26a","Type":"ContainerStarted","Data":"6834cb6c82ee54bb68960730a0ea96ae15deedb0c3407e4d913e2a4af22d36d6"} Oct 30 05:32:22.106408 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:22.105922 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4rx42" event={"ID":"16cdbe91-7a39-4ad3-8ef5-c738a995e26a","Type":"ContainerStarted","Data":"a27c4a4bc7c691ee23fc9deb5aeab5aecdaf42974f836aab773404b65c82fd6f"} Oct 30 05:32:22.108504 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:22.108477 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-phtr7" event={"ID":"794c8751-becf-4dc4-8574-282defa3eb69","Type":"ContainerStarted","Data":"252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e"} Oct 30 05:32:22.109958 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:22.109931 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wb5rz" event={"ID":"20949c65-577f-4bef-9494-b8ddec17c982","Type":"ContainerStarted","Data":"2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de"} Oct 30 05:32:22.152367 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:22.152300 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4rx42" podStartSLOduration=67.250875222 podStartE2EDuration="1m8.152280602s" totalImagesPullingTime="901.40538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:22.151749988 +0000 UTC m=+67.970740886" watchObservedRunningTime="2026-10-30 05:32:22.152280602 +0000 UTC m=+67.971271501" Oct 30 05:32:25.124096 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:25.124055 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-phtr7" event={"ID":"794c8751-becf-4dc4-8574-282defa3eb69","Type":"ContainerStarted","Data":"8ce628d30770700905386d2ca9d26661443972e16eeb6ee319324871e6ca1aca"} Oct 30 05:32:25.124517 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:25.124201 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:25.167893 ip-10-0-138-15 kubenswrapper[2583]: I1030 05:32:25.167837 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-phtr7" podStartSLOduration=68.154257709 podStartE2EDuration="1m11.167824049s" totalImagesPullingTime="3.01356634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:25.167381766 +0000 UTC m=+70.986372669" watchObservedRunningTime="2026-10-30 05:32:25.167824049 +0000 UTC m=+70.986814947" Oct 30 05:32:25.834206 ip-10-0-138-15 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:32:25.847638 ip-10-0-138-15 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:32:25.847822 ip-10-0-138-15 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:32:25.847882 ip-10-0-138-15 systemd[1]: kubelet.service: Consumed 4.130s CPU time. Oct 30 05:32:25.858775 ip-10-0-138-15 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:32:25.924499 ip-10-0-138-15 kubenswrapper[5980]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:32:25.924499 ip-10-0-138-15 kubenswrapper[5980]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:32:25.924499 ip-10-0-138-15 kubenswrapper[5980]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:32:25.924499 ip-10-0-138-15 kubenswrapper[5980]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:32:25.926609 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926515 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.926609 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926608 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.926609 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926612 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926615 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926618 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926622 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926625 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926627 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926630 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926633 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926635 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926638 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926641 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926643 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926646 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926649 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926651 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926654 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926656 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926659 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926664 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926667 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.926718 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926670 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926672 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926675 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926678 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926681 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926683 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926686 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926688 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926692 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926694 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926697 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926699 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926702 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926705 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926708 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926711 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926713 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926716 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926718 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926721 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.927209 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926724 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926726 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926729 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926731 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926734 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926736 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926739 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926741 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926744 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926746 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926749 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926752 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926755 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926757 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926760 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926763 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926766 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926769 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926772 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926774 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.927672 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926777 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926779 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926782 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926784 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926787 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926790 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926792 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926795 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926797 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926799 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926804 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926809 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926813 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926817 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926821 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926824 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926826 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926829 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.928150 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926831 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926834 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926836 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926839 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926841 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926843 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926846 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926848 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926851 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926853 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926856 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926859 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926862 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926864 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926867 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926869 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926872 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926875 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926878 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926880 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.928712 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926884 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926886 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926889 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926891 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926894 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.926897 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927711 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927719 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927723 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927727 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927731 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927735 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927738 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927741 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927743 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927746 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927749 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927751 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927754 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927756 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.929208 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927759 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927761 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927764 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927767 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927770 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927773 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927775 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927778 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927780 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927783 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927786 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927789 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927791 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927794 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927796 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927799 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927802 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927804 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927808 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927810 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.929679 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927813 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927816 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927819 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927821 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927823 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927826 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927828 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927831 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927833 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927836 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927838 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927841 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927843 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927846 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927849 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927851 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927854 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927856 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927859 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927861 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.930216 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927863 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927866 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927871 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927874 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927876 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927879 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927881 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927883 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927886 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927888 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927891 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927894 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927897 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927899 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927902 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927904 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927907 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927909 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927912 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.930681 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927915 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927919 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927923 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927926 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927929 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927931 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927934 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927936 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927939 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927941 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927944 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927946 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927948 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927951 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927954 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927959 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927961 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927963 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927966 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.931139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927968 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927971 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927973 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927975 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927978 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927980 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927983 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927986 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927989 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927991 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927994 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927997 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.927999 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928002 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928493 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928499 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928503 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928506 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928509 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928511 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.931750 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928514 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928517 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928519 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928522 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928525 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928527 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928530 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928532 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928536 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928539 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928541 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928544 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928548 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928551 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928554 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928557 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928560 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928562 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928566 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.932285 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928568 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928571 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928573 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928576 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928579 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928581 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928583 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928586 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928588 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928591 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928593 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928595 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928598 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928600 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928602 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928605 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928608 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928610 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928613 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928615 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.932790 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928618 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928621 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928625 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928628 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928631 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928633 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928636 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928638 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928641 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928643 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928645 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928648 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928654 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928656 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928659 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928661 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928664 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928666 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928668 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.933409 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928671 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928674 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928676 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928679 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928681 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928684 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928686 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928689 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928691 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928693 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928696 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928698 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928702 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928704 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928707 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928709 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928712 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928714 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928717 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.933890 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928720 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928723 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928725 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928728 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928730 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928733 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928737 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928740 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928742 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928745 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928747 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928750 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928752 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928755 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928757 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928760 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928762 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928765 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928774 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928779 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.934478 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928782 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928784 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.928787 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928886 5980 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928893 5980 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928899 5980 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928904 5980 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928908 5980 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928912 5980 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928915 5980 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928919 5980 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928923 5980 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928926 5980 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928929 5980 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928933 5980 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928936 5980 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928938 5980 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928941 5980 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928946 5980 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928949 5980 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928958 5980 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928962 5980 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928965 5980 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928968 5980 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:32:25.935245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928971 5980 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928975 5980 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928979 5980 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928982 5980 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928985 5980 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928988 5980 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928991 5980 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928994 5980 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.928997 5980 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929000 5980 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929003 5980 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929007 5980 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929010 5980 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929014 5980 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929017 5980 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929020 5980 flags.go:64] FLAG: --enable-server="true" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929024 5980 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929031 5980 flags.go:64] FLAG: --event-burst="100" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929034 5980 flags.go:64] FLAG: --event-qps="50" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929037 5980 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929040 5980 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929043 5980 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929047 5980 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929050 5980 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929053 5980 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:32:25.935835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929056 5980 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929060 5980 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929063 5980 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929066 5980 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929069 5980 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929073 5980 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929075 5980 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929078 5980 flags.go:64] FLAG: --feature-gates="" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929082 5980 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929085 5980 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929089 5980 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929092 5980 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929095 5980 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929098 5980 flags.go:64] FLAG: --help="false" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929101 5980 flags.go:64] FLAG: --hostname-override="ip-10-0-138-15.ec2.internal" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929119 5980 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929121 5980 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929124 5980 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929128 5980 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929132 5980 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929136 5980 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929139 5980 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929142 5980 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:32:25.936461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929144 5980 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929148 5980 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929151 5980 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929154 5980 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929156 5980 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929159 5980 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929162 5980 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929165 5980 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929167 5980 flags.go:64] FLAG: --lock-file="" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929170 5980 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929175 5980 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929178 5980 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929183 5980 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929186 5980 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929189 5980 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929192 5980 flags.go:64] FLAG: --logging-format="text" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929196 5980 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929199 5980 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929202 5980 flags.go:64] FLAG: --manifest-url="" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929205 5980 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929209 5980 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929212 5980 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929216 5980 flags.go:64] FLAG: --max-pods="110" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929219 5980 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929222 5980 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:32:25.937018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929225 5980 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929228 5980 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929231 5980 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929234 5980 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929238 5980 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929245 5980 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929248 5980 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929251 5980 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929254 5980 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929257 5980 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929260 5980 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929263 5980 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929266 5980 flags.go:64] FLAG: --port="10250" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929269 5980 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929271 5980 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00017c86398f70049" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929274 5980 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929277 5980 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929282 5980 flags.go:64] FLAG: --register-node="true" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929285 5980 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929291 5980 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929293 5980 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929297 5980 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929303 5980 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929307 5980 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929311 5980 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929314 5980 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:32:25.937620 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929316 5980 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929319 5980 flags.go:64] FLAG: --runonce="false" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929322 5980 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929325 5980 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929329 5980 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929331 5980 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929334 5980 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929337 5980 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929340 5980 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929344 5980 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929349 5980 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929352 5980 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929355 5980 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929358 5980 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929361 5980 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929369 5980 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929372 5980 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929378 5980 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929380 5980 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929383 5980 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929389 5980 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929393 5980 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929395 5980 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929399 5980 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929402 5980 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:32:25.938283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929405 5980 flags.go:64] FLAG: --v="2" Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929410 5980 flags.go:64] FLAG: --version="false" Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929414 5980 flags.go:64] FLAG: --vmodule="" Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929418 5980 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929429 5980 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929576 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929579 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929583 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929586 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929589 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929592 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929595 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929598 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929602 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929606 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929608 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929611 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929616 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929618 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929621 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.938909 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929623 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929626 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929629 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929631 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929634 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929636 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929639 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929641 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929644 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929646 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929650 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929652 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929655 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929657 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929660 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929663 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929665 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929668 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929671 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929673 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.939402 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929676 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929678 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929681 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929683 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929686 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929688 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929691 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929693 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929696 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929700 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929702 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929705 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929707 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929710 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929712 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929715 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929717 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929720 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929723 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929725 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.939895 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929728 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929731 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929735 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929737 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929740 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929742 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929744 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929748 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929752 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929755 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929760 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929763 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929766 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929768 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929771 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929774 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929776 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929779 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929781 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.940529 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929783 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929786 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929790 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929792 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929795 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929797 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929800 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929802 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929805 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929807 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929810 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929812 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929815 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929817 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929820 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929824 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929826 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929829 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929831 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.941175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929834 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929836 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929839 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929841 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929843 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929846 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929849 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929851 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929854 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929856 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929859 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929861 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929864 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.929889 5980 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929926 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929935 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.941700 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929939 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929941 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929944 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929947 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929950 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929952 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929955 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929958 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929960 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929963 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929965 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929968 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929971 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929974 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929977 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929979 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929982 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929984 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929987 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.942087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929989 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929992 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929994 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.929997 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930000 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930002 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930005 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930007 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930009 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930012 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930014 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930017 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930020 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930023 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930025 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930028 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930030 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930033 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930036 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930038 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.942610 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930040 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930043 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930045 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930048 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930052 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930055 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930060 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930063 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930065 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930068 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930070 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930073 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930077 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930080 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930082 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930084 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930088 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930090 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930093 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.943287 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930095 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930098 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930100 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930118 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930121 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930125 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930128 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930131 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930133 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930135 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930138 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930141 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930143 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930146 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930148 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930151 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930154 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930156 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930159 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930163 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.943731 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930165 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930168 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930171 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930173 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930176 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930178 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930181 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930183 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930185 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930188 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930191 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930193 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930196 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930198 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930201 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930203 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930206 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930209 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930214 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.944212 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930218 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930220 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930223 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930227 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930230 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930232 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.930235 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.934778 5980 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.934844 5980 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.934868 5980 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.934889 5980 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935064 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935072 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935077 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935135 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935141 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935146 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935151 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935156 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.944699 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935160 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935165 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935168 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935173 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935250 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935255 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935258 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935261 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935264 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935267 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935270 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935273 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935278 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935282 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935295 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935303 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935314 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935319 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935323 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.945175 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935326 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935334 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935433 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935436 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935439 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935442 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935444 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935447 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935450 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935452 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935455 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935458 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935460 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935463 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935466 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935468 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935470 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935473 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935476 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935478 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.945630 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935481 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935483 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935485 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935488 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935491 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935493 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935495 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935499 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935501 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935504 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935507 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935509 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935512 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935515 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935518 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935521 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935526 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935528 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935531 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935533 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.946087 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935536 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935538 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935541 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935544 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935546 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935550 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935554 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935558 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935561 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935564 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935567 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935569 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935572 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935575 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935577 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935580 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935582 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935584 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935587 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.946577 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935590 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935592 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935595 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935597 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935599 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935602 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935605 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935608 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935611 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935614 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935616 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935619 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935623 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935625 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935628 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935630 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935633 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935635 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935638 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.947028 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935641 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.935673 5980 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935846 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935851 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935855 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935857 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935860 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935863 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935866 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935869 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935871 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935874 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935877 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935879 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935881 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935885 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.947613 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935890 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935893 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935898 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935901 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935904 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935906 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935909 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935911 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935914 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935916 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935919 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935921 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935924 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935926 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935929 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935931 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935934 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935936 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935939 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935941 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.948056 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935943 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935946 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935948 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935951 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935954 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935957 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935960 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935962 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935965 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935967 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935969 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935972 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935974 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935977 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935980 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935982 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935984 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935987 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935989 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.948550 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935992 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935994 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935997 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.935999 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936002 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936005 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936007 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936010 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936012 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936015 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936017 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936020 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936022 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936025 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936027 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936030 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936033 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936035 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936039 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936041 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.949001 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936044 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936046 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936048 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936051 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936053 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936055 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936058 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936061 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936064 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936066 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936069 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936071 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936073 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936076 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936078 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936081 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936083 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936086 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936088 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.949492 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936090 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936093 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936095 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936098 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936101 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936129 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936132 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936135 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936138 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936142 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936145 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936148 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936151 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936154 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.936179 5980 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936213 5980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:25.949982 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936216 5980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936219 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936221 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936224 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936227 5980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936230 5980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936232 5980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936235 5980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936237 5980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936240 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936243 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936247 5980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936250 5980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936253 5980 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936256 5980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936258 5980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936261 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936263 5980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936266 5980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:25.950379 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936268 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936271 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936273 5980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936275 5980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936278 5980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936280 5980 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936283 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936286 5980 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936289 5980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936291 5980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936294 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936297 5980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936299 5980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936302 5980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936304 5980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936308 5980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936312 5980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936315 5980 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936318 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936320 5980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:25.950819 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936322 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936325 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936327 5980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936329 5980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936332 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936334 5980 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936336 5980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936339 5980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936341 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936344 5980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936346 5980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936349 5980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936351 5980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936353 5980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936356 5980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936358 5980 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936360 5980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936363 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936365 5980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936368 5980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:25.951299 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936370 5980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936373 5980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936375 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936378 5980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936381 5980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936383 5980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936385 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936388 5980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936391 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936393 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936396 5980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936398 5980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936401 5980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936403 5980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936406 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936408 5980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936410 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936413 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936415 5980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:25.951764 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936418 5980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936420 5980 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936422 5980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936425 5980 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936427 5980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936430 5980 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936432 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936434 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936437 5980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936439 5980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936442 5980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936444 5980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936447 5980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936449 5980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936452 5980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936454 5980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936456 5980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936459 5980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936462 5980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936464 5980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:25.952237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936467 5980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936469 5980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936474 5980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936477 5980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936480 5980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936483 5980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:25.936485 5980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.936541 5980 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.937591 5980 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.937649 5980 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.937830 5980 server.go:1014] "Starting client certificate rotation" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.937968 5980 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.938027 5980 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:26:14 +0000 UTC" deadline="2028-06-30 23:05:29.851584033 +0000 UTC" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.938059 5980 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14633h33m3.913528912s" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.938303 5980 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:32:25.952707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.939433 5980 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:32:25.953072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.940639 5980 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:32:25.953072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.942608 5980 log.go:25] "Validated CRI v1 image API" Oct 30 05:32:25.953072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.943341 5980 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:32:25.953072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.945412 5980 fs.go:135] Filesystem UUIDs: map[3f8b49d7-d447-46f1-a9f6-54085d4679fc:/dev/nvme0n1p4 7734fa11-071d-4d77-a8b9-b32636076296:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 05:32:25.953218 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.945427 5980 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de/userdata/shm major:0 minor:377 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0693da1f-8341-413f-b174-2575807a5a85/volumes/kubernetes.io~projected/kube-api-access-4lxk9:{mountpoint:/var/lib/kubelet/pods/0693da1f-8341-413f-b174-2575807a5a85/volumes/kubernetes.io~projected/kube-api-access-4lxk9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16cdbe91-7a39-4ad3-8ef5-c738a995e26a/volumes/kubernetes.io~projected/kube-api-access-pxfq8:{mountpoint:/var/lib/kubelet/pods/16cdbe91-7a39-4ad3-8ef5-c738a995e26a/volumes/kubernetes.io~projected/kube-api-access-pxfq8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16cdbe91-7a39-4ad3-8ef5-c738a995e26a/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/16cdbe91-7a39-4ad3-8ef5-c738a995e26a/volumes/kubernetes.io~secret/metrics-certs major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17d8104e-e85c-490a-855d-63af3e241f11/volumes/kubernetes.io~projected/kube-api-access-q4cwz:{mountpoint:/var/lib/kubelet/pods/17d8104e-e85c-490a-855d-63af3e241f11/volumes/kubernetes.io~projected/kube-api-access-q4cwz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20949c65-577f-4bef-9494-b8ddec17c982/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/20949c65-577f-4bef-9494-b8ddec17c982/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28230167-b221-4cb3-8601-fe8bd9605a73/volumes/kubernetes.io~projected/kube-api-access-g9vfz:{mountpoint:/var/lib/kubelet/pods/28230167-b221-4cb3-8601-fe8bd9605a73/volumes/kubernetes.io~projected/kube-api-access-g9vfz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/334ad7cd-4277-4572-b7e2-b0e2ececd5ff/volumes/kubernetes.io~projected/kube-api-access-2txdt:{mountpoint:/var/lib/kubelet/pods/334ad7cd-4277-4572-b7e2-b0e2ececd5ff/volumes/kubernetes.io~projected/kube-api-access-2txdt major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/334ad7cd-4277-4572-b7e2-b0e2ececd5ff/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/334ad7cd-4277-4572-b7e2-b0e2ececd5ff/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/config-reloader/2 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/prometheus/15 major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~projected/kube-api-access-87m9n:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~projected/kube-api-access-87m9n major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~projected/tls-assets major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/config major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/web-config major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~projected/kube-api-access-pl6c4:{mountpoint:/var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~projected/kube-api-access-pl6c4 major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/794c8751-becf-4dc4-8574-282defa3eb69/volumes/kubernetes.io~projected/kube-api-access-8ssgs:{mountpoint:/var/lib/kubelet/pods/794c8751-becf-4dc4-8574-282defa3eb69/volumes/kubernetes.io~projected/kube-api-access-8ssgs major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b010236-561c-4748-842a-648d3290ff2b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7b010236-561c-4748-842a-648d3290ff2b/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8885b33d-691c-4ba4-916d-c1e27fbd1cd3/volumes/kubernetes.io~projected/kube-api-access-nvdfb:{mountpoint:/var/lib/kubelet/pods/8885b33d-691c-4ba4-916d-c1e27fbd1cd3/volumes/kubernetes.io~projected/kube-api-access-nvdfb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~projected/kube-api-access-q4hdv:{mountpoint:/var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~projected/kube-api-access-q4hdv major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9cd7570-1937-4334-90a6-8c4d620a2216/volumes/kubernetes.io~projected/kube-api-access-mqbwb:{mountpoint:/var/lib/kubelet/pods/a9cd7570-1937-4334-90a6-8c4d620a2216/volumes/kubernetes.io~projected/kube-api-access-mqbwb major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9cd7570-1937-4334-90a6-8c4d620a2216/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a9cd7570-1937-4334-90a6-8c4d620a2216/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~projected/kube-api-access-5lkwg:{mountpoint:/var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~projected/kube-api-access-5lkwg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/volumes/kubernetes.io~projected/kube-api-access-n29gt:{mountpoint:/var/lib/kubelet/pods/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/volumes/kubernetes.io~projected/kube-api-access-n29gt major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volumes/kubernetes.io~projected/kube-api-access-szsn4:{mountpoint:/var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volumes/kubernetes.io~projected/kube-api-access-szsn4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5818964-b07b-4362-9eba-2458e3d58a97/volumes/kubernetes.io~projected/kube-api-access-qn8g9:{mountpoint:/var/lib/kubelet/pods/d5818964-b07b-4362-9eba-2458e3d58a97/volumes/kubernetes.io~projected/kube-api-access-qn8g9 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5818964-b07b-4362-9eba-2458e3d58a97/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d5818964-b07b-4362-9eba-2458e3d58a97/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eddf108a-bf5f-483b-ac39-1804ea6d8d76/volumes/kubernetes.io~projected/kube-api-access-sxvws:{mountpoint:/var/lib/kubelet/pods/eddf108a-bf5f-483b-ac39-1804ea6d8d76/volumes/kubernetes.io~projected/kube-api-access-sxvws major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~projected/kube-api-access-9wbvz:{mountpoint:/var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~projected/kube-api-access-9wbvz major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9766570-5a16-4a4a-aa4e-900bbbed459c/volumes/kubernetes.io~projected/kube-api-access-v7bjx:{mountpoint:/var/lib/kubelet/pods/f9766570-5a16-4a4a-aa4e-900bbbed459c/volumes/kubernetes.io~projected/kube-api-access-v7bjx major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8af29676ade2269a331a8ffffb40a5829759282289fb9ab2bbcde954d1e66e7e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5062873f6c18c21df0722b80569f8f916d4ad09525906d627d5a75fe2d0dcccc/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/0a9cc5a5fb84210bb4490fbc9f47b2f42386c397c3aad9c08f843df39745e08e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3a186c74d61bc2016780d430f72d918d4eaeffb6973f0bd5bcda38e13465b306/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2a65fbcd0af0e80429790fb50366baaefbd993b32f245d55911a3aca54f018ca/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/be93fb7aef83577758c9fd33f42ba3faa3f1e77b00cffa79772fb06413a5056a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d9b95eddef5d84d5abb12b6a0e909e218c5b6ac127ca1e45e3eed5ad0488d1fd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b2b8336263d651b5c105e016798495044b54cbb03bccefb1739e4bea7a72abae/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e4ad0ce4d21ea1c0369a9b03e89d87b70522985318d9538dd419acae999ac51e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a789db0f1648edce27d4517f6fef3f8b4d6f74852e26558758b0a853eff3fdfa/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/1e98eb75c2ccce3d3877f45513507aed1ef3c6f265e7cbd753e229cadaeb8989/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3446a567b8452a091ab7a1e19ff8a2a5710647323e20149882555d11315e5b2d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f2ab8c15a2ce6358d11bff5b1db3276d45c00ed3af8d6741ef423d369c7bbc6e/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/950a0716a653cbb06600db752bae9d0998000b62a1f9b363bc4182c1ba465dbd/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/81072030d9dcb6420291995f1df6c02ea6f5f5de371c60d60b257a0100b9d6d3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b0b512ce4b300234ae86e96f2922ddd6347b08d590cc6d8bf4ba570a06343ff9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a937576eb9a125927917c4051890198e1788d2a88924e5e0491ee217b75bc08a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/386aca4375515259fa2ac6f465bb1c34d6593ebab374985e3dea87e48aef355e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/88c9d048a3e611f94698a6250846143a59522727001972a911e03fd5586a67b0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/418214301b7dd10625b1da6486b99a355e39113267902893119c0d186c17655e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/7e72e77aacdcc2cb50b8dda62d3fa76c1cfc6571b9ed6d280acf1a1ba9c76168/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/9b9993747323b2ad65abb076a93e83e5206577ea34b918cb0288d36f6b8bf3dd/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/b6a3f86fc4f445bf235e83c2e91d50c2197f440191f81535babb6dd8417fea25/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1e5b15464c1ecbae11e9510976560b09364591fba7da1a7ad79d7b6940031e02/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/3f3c7947bd7d638e7da6cc8f6ac66acfad76d2d28ec424ceab3cd66d0bec8eed/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/b99388cff930bd7abc00547462d4e2be2776be468ab6579f7a4b5f2b00af051e/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/b17cd598567bee77e9e78e3149a2e08e5469b30cadd81a6119dbfe05fe47c7fd/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/86b00985245442d7563aa0db20f4f2720c439389ced147d8513777e2da3e293c/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/616b39325e61267f3a8a3ba7b6979eda12afe252e144751af59416ecd5483f68/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/906213f37c4870e6e6f193dcef26f3e63336308b742f1d999f93f2fec618ead8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/01a72db8376aad1ffa9644dd69a8427dec8b6eb853c1c20d3ee67ff296cea33e/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/f8fe1a23be1bd57c6c7c7cf03452e3fdc91576c8dbb6a53b979011fc082ce445/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/8a97b6874cb1b5c3c56478cd845172633e93a3f1c4495e2d9b632cfc2165cd9d/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/1fc9a54379ea15f3e34a709256a09d0f4a8a1695b1da527fcebc8105094ab8fc/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/ba7c2e8e1077e474e3d3a232434bd8be3cc642b48cee6cd4d52b97263fae40f9/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/710ddd979547c22fb9e7df76198293004fe6003ad9286849c3f52dd55b0684e7/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/3a8669d59280a0b267552642f3502eb4a7ee995747cc0ce3f380c42f5874136e/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/dbd9584d1ed036bd29cb1a8405aaff2f0f9604ff0ca59f23a3662dad2e98ef3a/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/69ff5a5a645c5ef6c4581f83a4cb6d67b8bfc3dbb0dfc15b0cbad2e14d20805b/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/5419831f03679721801eb0fb33a3dc74ceb767b470c4190168b89ad38287052c/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/f23b4ecf0554285a6b69c4532bb3332c057a01bdd215c7905b042bffd091fdba/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/780d0c3d616a202fa0149d6021378fcfca5f73ce4cb06bd689cd3445a9b1f9c2/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/9d7a86972f83e1c44a72a2e6be97936b0cbe8427beddb317920f3174d144bcbc/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/4e37d48f33a31f2dea19abb52e719a4c86716bb6fc9285a3f2fabbdd797e3ce4/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/8df06706a984aabe8bae7fb93654fe1e6455c206b1a138e3e67c5b04fac4de6b/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/08ccf62ae9b81700353edebb7f376c4b408e57d99c1540c06be83261464ce27e/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/de0ddc64b50053368d4da993f0e7c15feec36a66a7b496c81ba28e3ff2ee44ef/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/623ea59a039bbd6305e254b0a651979267322050ba1402ed59499bfd16991224/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/4e03871aa0b7e0727fa0252193f91bb78e2ed38dbd0e8948ed7efc6cedd46f52/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/9737ee5c75809012bd79149695b3976caec501baa0408ca3359c81007c3fb58b/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/f813e796f1fb688b0372cf8361a7d22e5d9e263fb502d863f436cd299f131ca1/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/573ed0d0a43f4da944f56547cb67cda52d1f0461dc867894fb3a4cd8637badfc/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/3ecf95ad956b7e29b91991a11621e562b01abcea16f98d4e9946d2a75945d692/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/5a06c39b2a13c9d3eb2d62ade7f49e327d9a8210f2b49cd74ffc51536f0aa8a3/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/9559b6f6416ba0092e6500601687b8fbe2180cf9973e47e6f58fbd886183c0f1/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/0b7aee746340d6bb452a7bab8e60b4a06655902f8803a88b7baacac4c3e4132d/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/093178f6c5faaed74d8175027496eab4d54398a671e91ff39ce2b8f19afb0bf9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6259ada59780576dd8805ed095b159cc608633a3f797073f8afef0b796f51fb4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a38832a57255022784770dfa3c78ba458b511fb6bcbf7ebbaf4d43f0e307d866/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c53d84ebad8afa19616e87e80cd84999c4908ed7a851df238904fe209e0ad256/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/19fe58d4b796362103d51352a2df6371ec02468014287d347b4286d88fb75ef3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6d0d38e9129668c09bd5d207af073f0ec98d51b24efd2f0f703beed43e14d4ee/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b6e52d29313f30700566612924f73daaf293f80e6efaf85eb09b49a2087938c0/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3cd438b52a3c81fee9af1fe1abe4a5f2d83f49d50144e8fb25dd498bc37f14a0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/9e9fd6cf4d96eb26c648033f3e27dbce42d759f25af1c6d61ac5624c3462617d/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cb84a3a1a4d76f62196ef53630ee2cc5fea1aabbad2d789afb6a01030fbfacfd/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/7e607d4d3e3d8dfde8b72f73a2b340b4e7e4c0aa63c44c89107ab7ee0dad7176/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 05:32:25.953451 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.952803 5980 manager.go:217] Machine: {Timestamp:2026-10-30 05:32:25.952336017 +0000 UTC m=+0.060375041 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101669 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ef2528acf69accb60f72c6adef420 SystemUUID:ec2ef252-8acf-69ac-cb60-f72c6adef420 BootID:ef338ee2-4cdf-4e7a-92d6-5fd8736a1f07 Filesystems:[{Device:/var/lib/kubelet/pods/a9cd7570-1937-4334-90a6-8c4d620a2216/volumes/kubernetes.io~projected/kube-api-access-mqbwb DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eddf108a-bf5f-483b-ac39-1804ea6d8d76/volumes/kubernetes.io~projected/kube-api-access-sxvws DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~projected/kube-api-access-q4hdv DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16cdbe91-7a39-4ad3-8ef5-c738a995e26a/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/252822995a739452aa469241e62197b590074482c135264069990a8829da2a2e/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b010236-561c-4748-842a-648d3290ff2b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48e8e0f35c58474f372c566ed4ca800960db8bd45b1d19100af2f97c128ae171/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48efee2b3606a784ff6b292ec31a334eca72a99736b988f1dbc7f9548b631b05/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f3b0d5d3882d6139160f01211afa56da6281dcf45fddde2f9079509102875de/userdata/shm DeviceMajor:0 DeviceMinor:377 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16cdbe91-7a39-4ad3-8ef5-c738a995e26a/volumes/kubernetes.io~projected/kube-api-access-pxfq8 DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/volumes/kubernetes.io~projected/kube-api-access-n29gt DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a8fc28bb2e5686d0ac96a2c7428f3538a4042b30b908eae06f3c5a316c28f89/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/794c8751-becf-4dc4-8574-282defa3eb69/volumes/kubernetes.io~projected/kube-api-access-8ssgs DeviceMajor:0 DeviceMinor:376 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9cd7570-1937-4334-90a6-8c4d620a2216/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecff709063c3261e0492f8b480fcb9c0f052112637ad7ebe56eeebb55be141e6/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volumes/kubernetes.io~projected/kube-api-access-szsn4 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/825e680252e6b62b7b76bac30ce08681c43d988d94d5be4d36cd94516cd8f247/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf2f1df0126c434bfabaf379268f280a297041128dafe9865de8cf72416568d5/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~projected/kube-api-access-87m9n DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b840af62b029a3a34f629377a86b27cadf434a2acbd812c315babbd288144a1/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98584416-9e1b-4358-8a58-23d4b123be72/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9766570-5a16-4a4a-aa4e-900bbbed459c/volumes/kubernetes.io~projected/kube-api-access-v7bjx DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/d5818964-b07b-4362-9eba-2458e3d58a97/volumes/kubernetes.io~projected/kube-api-access-qn8g9 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5818964-b07b-4362-9eba-2458e3d58a97/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b446dee0d4cf5d7182e63ccc7a27d800641c57a5623a331c5a7032a3352a8bb/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:326 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17d8104e-e85c-490a-855d-63af3e241f11/volumes/kubernetes.io~projected/kube-api-access-q4cwz DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa3959c0fcc760f53c56fa7580ef4bce8e30fca40d01ad096bf901669253d3d6/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15963b1da276cc9999fc62232aacb87a057e8712d16280e406c0f729ca981875/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/volumes/kubernetes.io~projected/kube-api-access-pl6c4 DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8885b33d-691c-4ba4-916d-c1e27fbd1cd3/volumes/kubernetes.io~projected/kube-api-access-nvdfb DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~projected/kube-api-access-5lkwg DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/334ad7cd-4277-4572-b7e2-b0e2ececd5ff/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5fcefcac041849e3ad41ed6994d2baaf6f39eb2bef94cb76823e3d0fb8d2265/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~projected/kube-api-access-9wbvz DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20949c65-577f-4bef-9494-b8ddec17c982/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:373 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:321 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0693da1f-8341-413f-b174-2575807a5a85/volumes/kubernetes.io~projected/kube-api-access-4lxk9 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/100a42ed62574d1b67e5555da8d042638ab0e23895446bb620a5e80458f2b997/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8b282a1-edcf-42d2-9f53-c995efc488cd/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:330 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/28230167-b221-4cb3-8601-fe8bd9605a73/volumes/kubernetes.io~projected/kube-api-access-g9vfz DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:332 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/319d7bfa4d458e23c8a9b3623ce84ebec287c38ec41883eb0ce768d38c8ed508/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c25f551c9de89ff59d4c431ae43b17bafbb6ac2a2d0260cff999f20fc88c790/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b718e7f2c484308b7aa87c2a50f6e17023fbc8e5e6d69567a2c71f9b917fc78/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5ebf1f331580f8313dfced37ebb91a1df1407b5bda95b6fffe558435c3a4b3c/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/334ad7cd-4277-4572-b7e2-b0e2ececd5ff/volumes/kubernetes.io~projected/kube-api-access-2txdt DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15963b1da276cc9 MacAddress:d2:4b:bf:9f:c1:93 Speed:10000 Mtu:8901} {Name:252822995a73945 MacAddress:0a:e0:fd:b4:7d:bc Speed:10000 Mtu:8901} {Name:2f3b0d5d3882d61 MacAddress:2e:48:19:e3:e6:6d Speed:10000 Mtu:8901} {Name:48e8e0f35c58474 MacAddress:b6:bc:48:06:d5:91 Speed:10000 Mtu:8901} {Name:48efee2b3606a78 MacAddress:0e:cf:d6:0c:55:26 Speed:10000 Mtu:8901} {Name:6a8fc28bb2e5686 MacAddress:b6:82:e9:3d:56:1b Speed:10000 Mtu:8901} {Name:858c5d8a8e7069f MacAddress:76:56:a0:33:86:33 Speed:10000 Mtu:8901} {Name:a5ebf1f331580f8 MacAddress:ca:59:e4:a4:87:3e Speed:10000 Mtu:8901} {Name:aa3959c0fcc760f MacAddress:16:70:85:fc:bb:75 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f5:1d:ee:cf Speed:0 Mtu:9001} {Name:br-int MacAddress:82:83:35:c5:20:dd Speed:0 Mtu:8901} {Name:c5fcefcac041849 MacAddress:5e:a5:13:de:77:fe Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f5:1d:ee:cf Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:a3:39:17:83:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:de:2e:72:91:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:32:25.953724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.953481 5980 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:32:25.953724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.953559 5980 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:32:25.954239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.954101 5980 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:32:25.954908 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.954153 5980 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:32:25.955046 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955034 5980 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:32:25.955079 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955050 5980 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:32:25.955079 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955063 5980 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:32:25.955215 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955085 5980 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:32:25.955215 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955214 5980 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:32:25.955405 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955394 5980 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:32:25.955522 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955513 5980 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:32:25.955562 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955553 5980 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:32:25.955594 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955568 5980 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:32:25.955594 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955579 5980 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:32:25.955594 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955588 5980 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:32:25.955681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.955596 5980 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:32:25.956777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.956752 5980 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:32:25.956777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.956777 5980 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:32:25.957341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.957327 5980 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:32:25.960009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.959993 5980 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:32:25.960009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960008 5980 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:32:25.960143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960122 5980 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:32:25.960335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960323 5980 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:32:25.960371 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960353 5980 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:32:25.960453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960444 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960459 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960465 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960471 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960476 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960482 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960488 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:32:25.960494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960494 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:32:25.960877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960501 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:32:25.960877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960507 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:32:25.960877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960515 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:32:25.960877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960524 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:32:25.960877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960553 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:32:25.960877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.960559 5980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:32:25.963936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.963923 5980 server.go:1263] "Started kubelet" Oct 30 05:32:25.964037 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.964002 5980 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:32:25.964087 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.964021 5980 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:32:25.964509 ip-10-0-138-15 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:32:25.965457 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.965435 5980 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:32:25.966197 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.966174 5980 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:32:25.966569 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.966523 5980 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:32:25.969236 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969132 5980 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:32:25.969341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969316 5980 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:32:25.969341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969333 5980 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:32:25.969615 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969510 5980 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:32:25.969615 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969195 5980 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:32:25.969735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969677 5980 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:26:16 +0000 UTC" deadline="2028-07-25 13:27:17.302139252 +0000 UTC" Oct 30 05:32:25.969735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.969726 5980 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15223h54m51.332417386s" Oct 30 05:32:25.970029 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:25.969848 5980 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-15.ec2.internal\" not found" Oct 30 05:32:25.970865 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.970609 5980 factory.go:55] Registering systemd factory Oct 30 05:32:25.970865 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.970630 5980 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:32:25.973786 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973745 5980 factory.go:153] Registering CRI-O factory Oct 30 05:32:25.973786 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973764 5980 factory.go:223] Registration of the crio container factory successfully Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973821 5980 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973747 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693da1f-8341-413f-b174-2575807a5a85" volumeName="kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-binary-copy" seLinuxMountContext="" Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973842 5980 factory.go:103] Registering Raw factory Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973857 5980 manager.go:1199] Started watching for new ooms in manager Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973854 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="334ad7cd-4277-4572-b7e2-b0e2ececd5ff" volumeName="kubernetes.io/secret/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-metrics-tls" seLinuxMountContext="" Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973872 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973889 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973902 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9cd7570-1937-4334-90a6-8c4d620a2216" volumeName="kubernetes.io/projected/a9cd7570-1937-4334-90a6-8c4d620a2216-kube-api-access-mqbwb" seLinuxMountContext="" Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973914 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b77f59d5-e2ae-46a9-b3b5-6333ec8ca595" volumeName="kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-tuned" seLinuxMountContext="" Oct 30 05:32:25.973928 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973925 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b" volumeName="kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-env-overrides" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973937 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693da1f-8341-413f-b174-2575807a5a85" volumeName="kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973953 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b" volumeName="kubernetes.io/secret/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973967 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973978 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" volumeName="kubernetes.io/projected/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-kube-api-access-pxfq8" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.973990 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="334ad7cd-4277-4572-b7e2-b0e2ececd5ff" volumeName="kubernetes.io/empty-dir/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-tmp-dir" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974002 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974015 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974031 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" volumeName="kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974044 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" volumeName="kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-tls" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974056 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" volumeName="kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974067 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b" volumeName="kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-config" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974080 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974093 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974125 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" volumeName="kubernetes.io/empty-dir/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-textfile" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974139 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b" volumeName="kubernetes.io/projected/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-kube-api-access-szsn4" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974151 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddf108a-bf5f-483b-ac39-1804ea6d8d76" volumeName="kubernetes.io/projected/eddf108a-bf5f-483b-ac39-1804ea6d8d76-kube-api-access-sxvws" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974162 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b282a1-edcf-42d2-9f53-c995efc488cd" volumeName="kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974175 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974187 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98584416-9e1b-4358-8a58-23d4b123be72" volumeName="kubernetes.io/projected/98584416-9e1b-4358-8a58-23d4b123be72-kube-api-access-q4hdv" seLinuxMountContext="" Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974200 5980 manager.go:319] Starting recovery of all containers Oct 30 05:32:25.974417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974199 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b77f59d5-e2ae-46a9-b3b5-6333ec8ca595" volumeName="kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-tmp" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974464 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff6f637-e30a-47b8-88fe-a5aa3dfd5e02" volumeName="kubernetes.io/secret/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974500 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b282a1-edcf-42d2-9f53-c995efc488cd" volumeName="kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974518 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b282a1-edcf-42d2-9f53-c995efc488cd" volumeName="kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974533 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974546 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b010236-561c-4748-842a-648d3290ff2b" volumeName="kubernetes.io/secret/7b010236-561c-4748-842a-648d3290ff2b-agent-certs" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974559 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bff6f637-e30a-47b8-88fe-a5aa3dfd5e02" volumeName="kubernetes.io/projected/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-kube-api-access-n29gt" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974573 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b282a1-edcf-42d2-9f53-c995efc488cd" volumeName="kubernetes.io/empty-dir/f8b282a1-edcf-42d2-9f53-c995efc488cd-volume-directive-shadow" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974585 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16cdbe91-7a39-4ad3-8ef5-c738a995e26a" volumeName="kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974593 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28230167-b221-4cb3-8601-fe8bd9605a73" volumeName="kubernetes.io/projected/28230167-b221-4cb3-8601-fe8bd9605a73-kube-api-access-g9vfz" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974606 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974614 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974622 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8885b33d-691c-4ba4-916d-c1e27fbd1cd3" volumeName="kubernetes.io/projected/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-kube-api-access-nvdfb" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974630 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98584416-9e1b-4358-8a58-23d4b123be72" volumeName="kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974638 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9cd7570-1937-4334-90a6-8c4d620a2216" volumeName="kubernetes.io/secret/a9cd7570-1937-4334-90a6-8c4d620a2216-cert" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974647 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693da1f-8341-413f-b174-2575807a5a85" volumeName="kubernetes.io/projected/0693da1f-8341-413f-b174-2575807a5a85-kube-api-access-4lxk9" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974668 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974677 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="794c8751-becf-4dc4-8574-282defa3eb69" volumeName="kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974684 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8885b33d-691c-4ba4-916d-c1e27fbd1cd3" volumeName="kubernetes.io/empty-dir/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-tmp-dir" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974700 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b" volumeName="kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974708 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17d8104e-e85c-490a-855d-63af3e241f11" volumeName="kubernetes.io/projected/17d8104e-e85c-490a-855d-63af3e241f11-kube-api-access-q4cwz" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974715 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28230167-b221-4cb3-8601-fe8bd9605a73" volumeName="kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-multus-daemon-config" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974731 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974739 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" volumeName="kubernetes.io/projected/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-kube-api-access-pl6c4" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974748 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98584416-9e1b-4358-8a58-23d4b123be72" volumeName="kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974766 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5818964-b07b-4362-9eba-2458e3d58a97" volumeName="kubernetes.io/configmap/d5818964-b07b-4362-9eba-2458e3d58a97-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974774 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b282a1-edcf-42d2-9f53-c995efc488cd" volumeName="kubernetes.io/projected/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-api-access-9wbvz" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974800 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0693da1f-8341-413f-b174-2575807a5a85" volumeName="kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974815 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="334ad7cd-4277-4572-b7e2-b0e2ececd5ff" volumeName="kubernetes.io/configmap/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-config-volume" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974823 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="334ad7cd-4277-4572-b7e2-b0e2ececd5ff" volumeName="kubernetes.io/projected/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-kube-api-access-2txdt" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974837 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b010236-561c-4748-842a-648d3290ff2b" volumeName="kubernetes.io/configmap/7b010236-561c-4748-842a-648d3290ff2b-konnectivity-ca" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974853 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28230167-b221-4cb3-8601-fe8bd9605a73" volumeName="kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-cni-binary-copy" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974869 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" volumeName="kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974877 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5818964-b07b-4362-9eba-2458e3d58a97" volumeName="kubernetes.io/empty-dir/d5818964-b07b-4362-9eba-2458e3d58a97-data-volume" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974895 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddf108a-bf5f-483b-ac39-1804ea6d8d76" volumeName="kubernetes.io/configmap/eddf108a-bf5f-483b-ac39-1804ea6d8d76-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974915 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974922 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974942 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974950 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b77f59d5-e2ae-46a9-b3b5-6333ec8ca595" volumeName="kubernetes.io/projected/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-kube-api-access-5lkwg" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974956 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5818964-b07b-4362-9eba-2458e3d58a97" volumeName="kubernetes.io/projected/d5818964-b07b-4362-9eba-2458e3d58a97-kube-api-access-qn8g9" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974987 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5818964-b07b-4362-9eba-2458e3d58a97" volumeName="kubernetes.io/secret/d5818964-b07b-4362-9eba-2458e3d58a97-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.974994 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975001 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975030 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975039 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20949c65-577f-4bef-9494-b8ddec17c982" volumeName="kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975051 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9766570-5a16-4a4a-aa4e-900bbbed459c" volumeName="kubernetes.io/projected/f9766570-5a16-4a4a-aa4e-900bbbed459c-kube-api-access-v7bjx" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975082 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abeb78e-0d20-44fe-a201-5855719acd9b" volumeName="kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975090 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98584416-9e1b-4358-8a58-23d4b123be72" volumeName="kubernetes.io/configmap/98584416-9e1b-4358-8a58-23d4b123be72-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975098 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8b282a1-edcf-42d2-9f53-c995efc488cd" volumeName="kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975162 5980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9766570-5a16-4a4a-aa4e-900bbbed459c" volumeName="kubernetes.io/configmap/f9766570-5a16-4a4a-aa4e-900bbbed459c-serviceca" seLinuxMountContext="" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975172 5980 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:32:25.975247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:25.975177 5980 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:32:26.003545 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.003490 5980 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:32:26.005296 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.005262 5980 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:32:26.005296 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.005292 5980 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:32:26.005501 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.005316 5980 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:32:26.005501 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:26.005355 5980 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:32:26.015699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.015646 5980 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="570486ff266db57732feed3ff8350508da23a3fa08e69c43c17576b4ebeb965e" exitCode=0 Oct 30 05:32:26.015699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.015695 5980 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="736b381e82ba0ddb679aca1bd2645a9c7e268bc84b8506b347d3749eaf2e1ba2" exitCode=0 Oct 30 05:32:26.015699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.015706 5980 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="85b877f536d3283901c73ca98a6675ccb80e0cdd82b68728fe4cad2aec925adf" exitCode=0 Oct 30 05:32:26.015919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.015720 5980 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="0871a1c530bda2255403a1af15ee306d7a3055d3e891bc865d5bf881fe41ae84" exitCode=0 Oct 30 05:32:26.015919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.015730 5980 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="a7d6ac2cacfb08c5cde909ae29d7540655ba74c05119727dc7bb3cf63e3412e8" exitCode=0 Oct 30 05:32:26.015919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.015740 5980 generic.go:356] "Generic (PLEG): container finished" podID="0693da1f-8341-413f-b174-2575807a5a85" containerID="a0b1ca3f72ea87186bf51607fcd5a51d38cf07d91706b3c725cc73fb1224f74f" exitCode=0 Oct 30 05:32:26.017754 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.017730 5980 generic.go:356] "Generic (PLEG): container finished" podID="a5d26ba6ff099574f4393ea9d38dc6f8" containerID="dc88da18c05c62df39a44ce17328550c8eca04b4c7c2629067863f46da5a840b" exitCode=0 Oct 30 05:32:26.020368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.020350 5980 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:32:26.021623 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.021599 5980 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:26.021808 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.021777 5980 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:26.029958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.029929 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" exitCode=0 Oct 30 05:32:26.032021 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.031995 5980 generic.go:356] "Generic (PLEG): container finished" podID="714aff30-d1b2-4ed7-9a37-e7674c4c2ad3" containerID="9d3a59d1c3bd2bf91d2b308d93b0f8518d0bd7a2d8c0f955cea8b57d6dd2c850" exitCode=0 Oct 30 05:32:26.037261 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.037237 5980 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:26.045754 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.045730 5980 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:26.072999 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.072978 5980 manager.go:324] Recovery completed Oct 30 05:32:26.096530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096440 5980 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:32:26.096530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096460 5980 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:32:26.096530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096484 5980 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096643 5980 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096659 5980 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096683 5980 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096692 5980 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096699 5980 policy_none.go:50] "Start" Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096710 5980 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:32:26.096764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096720 5980 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:32:26.097050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096859 5980 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:32:26.097050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096866 5980 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:32:26.097050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.096872 5980 policy_none.go:44] "Start" Oct 30 05:32:26.101462 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.101428 5980 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:32:26.105911 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:26.105893 5980 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 05:32:26.106000 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.105980 5980 manager.go:341] "Starting Device Plugin manager" Oct 30 05:32:26.106594 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:26.106003 5980 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:32:26.106631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.106599 5980 server.go:87] "Starting device plugin registration server" Oct 30 05:32:26.106882 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.106872 5980 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:32:26.106921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.106884 5980 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:32:26.107016 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.106991 5980 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:32:26.107181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.107095 5980 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:32:26.107181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.107150 5980 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:32:26.107181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.107155 5980 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:32:26.107330 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.107209 5980 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:32:26.107140998Z","UUID":"08e372e8-857b-495d-bbda-b1d46661d975","Handler":null,"Name":"","Endpoint":""} Oct 30 05:32:26.108632 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.108615 5980 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:32:26.108721 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.108642 5980 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:32:26.108931 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:26.108911 5980 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:32:26.208376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.208343 5980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:32:26.210910 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.210890 5980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:32:26.210996 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.210924 5980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:32:26.210996 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.210934 5980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-15.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:32:26.210996 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.210977 5980 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.248241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.248210 5980 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.248399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.248294 5980 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.306215 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.306161 5980 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal"] Oct 30 05:32:26.306906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.306888 5980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.306988 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.306888 5980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.307077 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.306858 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" event={"ID":"a5d26ba6ff099574f4393ea9d38dc6f8","Type":"ContainerStarted","Data":"db8b4253cf134b457d5c0fadd89fe1f0c40a9dab653ecf08942268f9952088b0"} Oct 30 05:32:26.307157 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.307097 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" event={"ID":"a5d26ba6ff099574f4393ea9d38dc6f8","Type":"ContainerDied","Data":"dc88da18c05c62df39a44ce17328550c8eca04b4c7c2629067863f46da5a840b"} Oct 30 05:32:26.307157 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.307145 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" event={"ID":"a5d26ba6ff099574f4393ea9d38dc6f8","Type":"ContainerStarted","Data":"cc9d81daebf6207ae6d5e3afe7187f10e15131c0c51ad17f4a227397fa1fe5f6"} Oct 30 05:32:26.307247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.307223 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" event={"ID":"eddf44eee240eccf9696b703c5918959","Type":"ContainerStarted","Data":"da33e430f75db92257e6c4558cf2a07e3d58bf28c975663dcd2207e1b5b44e2c"} Oct 30 05:32:26.307247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.307237 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" event={"ID":"eddf44eee240eccf9696b703c5918959","Type":"ContainerStarted","Data":"743b6af1401dd73fb88e6c2fd01d3f869be26164f815bfb73d2ced67b5cbb6d8"} Oct 30 05:32:26.331502 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.331471 5980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:26.331614 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:26.331530 5980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.332216 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.332198 5980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:26.332258 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:26.332240 5980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.421658 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.421582 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.421658 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.421614 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.421658 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.421644 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddf44eee240eccf9696b703c5918959-config\") pod \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" (UID: \"eddf44eee240eccf9696b703c5918959\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.522233 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.522193 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.522233 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.522232 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.522417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.522252 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddf44eee240eccf9696b703c5918959-config\") pod \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" (UID: \"eddf44eee240eccf9696b703c5918959\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.522417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.522302 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eddf44eee240eccf9696b703c5918959-config\") pod \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" (UID: \"eddf44eee240eccf9696b703c5918959\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.522417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.522311 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.522417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.522311 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d26ba6ff099574f4393ea9d38dc6f8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" (UID: \"a5d26ba6ff099574f4393ea9d38dc6f8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:26.956008 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:26.955972 5980 apiserver.go:52] "Watching apiserver" Oct 30 05:32:27.010549 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.010510 5980 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:32:27.011132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.011090 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s","openshift-multus/multus-additional-cni-plugins-pcx4f","openshift-cluster-node-tuning-operator/tuned-fzbh9","openshift-image-registry/node-ca-8mh7z","openshift-network-diagnostics/network-check-target-phtr7","kube-system/konnectivity-agent-sxrx6","openshift-ingress-canary/ingress-canary-b896w","openshift-insights/insights-runtime-extractor-ctntz","openshift-monitoring/node-exporter-68zhv","openshift-multus/network-metrics-daemon-4rx42","kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z","openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq","openshift-dns/dns-default-rm8hh","openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk","openshift-network-operator/iptables-alerter-lp49f","kube-system/global-pull-secret-syncer-wb5rz","openshift-ovn-kubernetes/ovnkube-node-cr5xj","openshift-dns/node-resolver-q2qpx","openshift-multus/multus-kcw92","openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:27.031348 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031276 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:32:27.031348 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031272 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:32:27.031348 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031283 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.031613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031410 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:32:27.031759 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031741 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.031884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031782 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:32:27.031884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031796 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nr89r\"" Oct 30 05:32:27.031884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031858 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:32:27.031884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031874 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.032264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.031879 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.032264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032028 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k22sc\"" Oct 30 05:32:27.032264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032055 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.032264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032058 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032272 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032299 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032339 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf7fl\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032345 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032339 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032376 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8jj4\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032387 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032377 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2ztbt\"" Oct 30 05:32:27.032459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032436 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:32:27.032707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032491 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jtd4z\"" Oct 30 05:32:27.032707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.032553 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8j26\"" Oct 30 05:32:27.064872 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.064845 5980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:27.064872 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.064857 5980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:27.071681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.071658 5980 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:32:27.091581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.091556 5980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:27.091682 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:27.091602 5980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-15.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" Oct 30 05:32:27.091853 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.091837 5980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:27.091928 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:27.091882 5980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-15.ec2.internal" Oct 30 05:32:27.125699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125662 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-netns\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.125699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125699 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-bin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.125949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125735 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.125949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125791 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbwb\" (UniqueName: \"kubernetes.io/projected/a9cd7570-1937-4334-90a6-8c4d620a2216-kube-api-access-mqbwb\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:27.125949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125871 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovn-node-metrics-cert\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.125949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125924 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-hosts-file\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.126099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125950 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6c4\" (UniqueName: \"kubernetes.io/projected/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-kube-api-access-pl6c4\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.126099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125978 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.126099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.125976 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.126099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126018 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-cnibin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.126250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126090 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-sys-fs\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.126250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126157 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-host\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.126250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126208 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-cni-binary-copy\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.126250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126234 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-k8s-cni-cncf-io\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.126250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126250 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-sys\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.126407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126271 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-accelerators-collector-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.126407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126316 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.126407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126355 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.126407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126400 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-dbus\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126431 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-cni-binary-copy\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126435 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5818964-b07b-4362-9eba-2458e3d58a97-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126460 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8g9\" (UniqueName: \"kubernetes.io/projected/d5818964-b07b-4362-9eba-2458e3d58a97-kube-api-access-qn8g9\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126515 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-tmp-dir\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126550 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkwg\" (UniqueName: \"kubernetes.io/projected/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-kube-api-access-5lkwg\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126575 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-tmp-dir\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126581 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-multus\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.126616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126613 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-multus-daemon-config\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126646 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126668 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126732 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126754 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txdt\" (UniqueName: \"kubernetes.io/projected/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-kube-api-access-2txdt\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126773 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126796 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126823 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szsn4\" (UniqueName: \"kubernetes.io/projected/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-kube-api-access-szsn4\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126848 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-system-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126873 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5818964-b07b-4362-9eba-2458e3d58a97-crio-socket\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126913 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-wtmp\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126946 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbvz\" (UniqueName: \"kubernetes.io/projected/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-api-access-9wbvz\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126970 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-lib-modules\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.126985 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-socket-dir-parent\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127000 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-netns\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127134 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5818964-b07b-4362-9eba-2458e3d58a97-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127172 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-etc-kubernetes\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127232 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bjx\" (UniqueName: \"kubernetes.io/projected/f9766570-5a16-4a4a-aa4e-900bbbed459c-kube-api-access-v7bjx\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127283 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-system-cni-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127399 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-cnibin\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127441 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-slash\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127466 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-conf-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127492 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127510 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-sys\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.127782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127547 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.127782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127605 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.127782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127641 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.127782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127682 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-multus-certs\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.127782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127721 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.127782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127751 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98584416-9e1b-4358-8a58-23d4b123be72-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.127938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127803 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-metrics-tls\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.127938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127839 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9766570-5a16-4a4a-aa4e-900bbbed459c-host\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.127938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127908 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9766570-5a16-4a4a-aa4e-900bbbed459c-serviceca\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.128020 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127954 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-etc-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.128020 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.127997 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:27.128093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128045 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-textfile\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.128093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128083 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-metrics-client-ca\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.128202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128162 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-textfile\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.128241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128214 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hdv\" (UniqueName: \"kubernetes.io/projected/98584416-9e1b-4358-8a58-23d4b123be72-kube-api-access-q4hdv\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.128306 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128288 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-systemd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.128363 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128321 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-socket-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.128393 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128383 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:27.128450 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128436 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-run\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.128481 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128468 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.128531 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128516 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxfq8\" (UniqueName: \"kubernetes.io/projected/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-kube-api-access-pxfq8\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:27.128582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128569 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b010236-561c-4748-842a-648d3290ff2b-konnectivity-ca\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:32:27.128615 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128584 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9766570-5a16-4a4a-aa4e-900bbbed459c-serviceca\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.128615 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128599 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.128675 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128629 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.128675 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128653 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-tmp\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.128733 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128683 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.128733 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128713 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b010236-561c-4748-842a-648d3290ff2b-agent-certs\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:32:27.128828 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128743 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29gt\" (UniqueName: \"kubernetes.io/projected/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-kube-api-access-n29gt\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:27.128828 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128754 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-tmp\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.128828 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128797 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-conf\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.128828 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128825 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.128963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128865 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cd7570-1937-4334-90a6-8c4d620a2216-cert\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:27.128963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128911 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-ovn\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.128963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128942 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-config\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.128963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128958 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128970 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5818964-b07b-4362-9eba-2458e3d58a97-data-volume\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.128996 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129025 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysconfig\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129043 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b010236-561c-4748-842a-648d3290ff2b-agent-certs\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129052 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-kubernetes\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129047 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5818964-b07b-4362-9eba-2458e3d58a97-data-volume\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.129099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129084 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.129316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129143 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-var-lib-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129173 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-registration-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.129316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129197 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-root\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.129316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129243 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.129316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129274 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.129316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129307 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eddf108a-bf5f-483b-ac39-1804ea6d8d76-host-slash\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.129468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129332 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-tls\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.129468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129357 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-kubelet\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129379 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-script-lib\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129409 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.129468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129430 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.129468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129445 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-node-log\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129621 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129462 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxk9\" (UniqueName: \"kubernetes.io/projected/0693da1f-8341-413f-b174-2575807a5a85-kube-api-access-4lxk9\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.129621 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129575 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-bin\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129638 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-env-overrides\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129687 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cwz\" (UniqueName: \"kubernetes.io/projected/17d8104e-e85c-490a-855d-63af3e241f11-kube-api-access-q4cwz\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.129714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129709 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-netd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129728 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.129799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129747 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxvws\" (UniqueName: \"kubernetes.io/projected/eddf108a-bf5f-483b-ac39-1804ea6d8d76-kube-api-access-sxvws\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.129799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129766 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-tuned\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.129799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129791 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-config-volume\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.129931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129814 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.129931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129830 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.129931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129833 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-tuned\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.129931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129847 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.129931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129872 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-var-lib-kubelet\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.129931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129902 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130101 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129934 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130101 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.129945 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-systemd-units\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.130101 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130042 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-log-socket\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.130101 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130076 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vfz\" (UniqueName: \"kubernetes.io/projected/28230167-b221-4cb3-8601-fe8bd9605a73-kube-api-access-g9vfz\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.130258 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130147 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eddf108a-bf5f-483b-ac39-1804ea6d8d76-iptables-alerter-script\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.130258 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130178 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-modprobe-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.130325 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130291 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130354 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130332 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eddf108a-bf5f-483b-ac39-1804ea6d8d76-iptables-alerter-script\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.130354 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130339 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-hostroot\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.130434 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130366 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:27.130522 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130498 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdfb\" (UniqueName: \"kubernetes.io/projected/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-kube-api-access-nvdfb\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.130575 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130539 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-systemd\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.130575 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130567 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-tmp-dir\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.130683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130606 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-metrics-certs\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:27.130683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130611 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130623 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.130683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130643 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-tmp-dir\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.130683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130650 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130679 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130712 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f8b282a1-edcf-42d2-9f53-c995efc488cd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130739 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-device-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130757 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130770 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-kubelet\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130784 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f8b282a1-edcf-42d2-9f53-c995efc488cd-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130839 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130845 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130848 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130869 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130904 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130922 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.130946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130952 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.131386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130975 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-kubelet-config\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.131386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.130996 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.131386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131026 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.131386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131160 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0693da1f-8341-413f-b174-2575807a5a85-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.131386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131219 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.131386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131373 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.131615 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131601 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:32:27.131671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131654 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:32:27.131736 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131688 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:32:27.131859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131842 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.131918 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131867 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:32:27.131918 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131873 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.131918 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.131904 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:32:27.132197 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132179 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:32:27.132320 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132187 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:32:27.132441 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132425 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:32:27.132552 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132439 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:32:27.132552 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132453 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:32:27.132552 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132454 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-t7gdv\"" Oct 30 05:32:27.132643 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132615 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:32:27.132721 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132709 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:32:27.132886 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132874 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:32:27.132934 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.132923 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:32:27.133121 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133097 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.133184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133142 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:32:27.133184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133147 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.133184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133157 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:32:27.133184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133165 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.133390 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133369 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133400 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133407 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133419 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wdgdh\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133411 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133420 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-x96j8\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133411 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133430 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7pqs8\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133449 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:32:27.133510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.133477 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:32:27.136780 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.136755 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5818964-b07b-4362-9eba-2458e3d58a97-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.136878 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.136806 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-accelerators-collector-config\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.136878 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.136847 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.136964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.136951 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28230167-b221-4cb3-8601-fe8bd9605a73-multus-daemon-config\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.137037 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.137024 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.138008 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.137995 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5818964-b07b-4362-9eba-2458e3d58a97-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.139195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139179 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.139249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139183 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cd7570-1937-4334-90a6-8c4d620a2216-cert\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:27.139249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139182 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-config\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.139310 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139296 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.139342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139302 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.139929 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139911 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mkgv\"" Oct 30 05:32:27.140056 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139962 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:32:27.140056 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139971 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:32:27.140056 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.139973 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:32:27.140056 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140024 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2eaov2pk6s7tl\"" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140083 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-env-overrides\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140101 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-tls\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140123 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140149 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140222 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjhr7\"" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140233 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140242 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140224 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140258 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vg6rp\"" Oct 30 05:32:27.140305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140291 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xpx6p\"" Oct 30 05:32:27.140660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140323 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.140660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140369 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-config-volume\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.140660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.140553 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.141717 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.141699 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.141755 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.141722 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.141755 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.141706 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.144587 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.144532 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pqn7\"" Oct 30 05:32:27.144587 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.144532 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wks87\"" Oct 30 05:32:27.144722 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.144659 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:32:27.145190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.145167 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvn8d\"" Oct 30 05:32:27.145394 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.145372 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bmrwn\"" Oct 30 05:32:27.145484 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.145460 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 05:32:27.145735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.145717 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 05:32:27.145820 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.145802 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:32:27.146422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.146248 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:32:27.146422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.146274 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lg7qg\"" Oct 30 05:32:27.146422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.146281 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 05:32:27.146422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.146252 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:32:27.146422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.146345 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovn-node-metrics-cert\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.146698 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.146579 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 05:32:27.147022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.147002 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.147241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.147222 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.148498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.148480 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-metrics-tls\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.148589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.148518 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.148792 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.148761 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:32:27.148860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.148761 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:27.148947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.148924 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.149436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.149382 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txdt\" (UniqueName: \"kubernetes.io/projected/334ad7cd-4277-4572-b7e2-b0e2ececd5ff-kube-api-access-2txdt\") pod \"dns-default-rm8hh\" (UID: \"334ad7cd-4277-4572-b7e2-b0e2ececd5ff\") " pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:27.149508 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.149452 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6c4\" (UniqueName: \"kubernetes.io/projected/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-kube-api-access-pl6c4\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.149569 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.149415 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbwb\" (UniqueName: \"kubernetes.io/projected/a9cd7570-1937-4334-90a6-8c4d620a2216-kube-api-access-mqbwb\") pod \"ingress-canary-b896w\" (UID: \"a9cd7570-1937-4334-90a6-8c4d620a2216\") " pod="openshift-ingress-canary/ingress-canary-b896w" Oct 30 05:32:27.149796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.149775 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkwg\" (UniqueName: \"kubernetes.io/projected/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-kube-api-access-5lkwg\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.150096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150074 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-ovnkube-script-lib\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.150189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150154 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:32:27.150189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150158 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8b282a1-edcf-42d2-9f53-c995efc488cd-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.150323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150304 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.150618 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150599 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bjx\" (UniqueName: \"kubernetes.io/projected/f9766570-5a16-4a4a-aa4e-900bbbed459c-kube-api-access-v7bjx\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.150698 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150679 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98584416-9e1b-4358-8a58-23d4b123be72-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.150755 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150684 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szsn4\" (UniqueName: \"kubernetes.io/projected/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-kube-api-access-szsn4\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.150809 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150788 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.150860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150804 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b010236-561c-4748-842a-648d3290ff2b-konnectivity-ca\") pod \"konnectivity-agent-sxrx6\" (UID: \"7b010236-561c-4748-842a-648d3290ff2b\") " pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:32:27.150860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150800 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-metrics-client-ca\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.150961 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150860 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/98584416-9e1b-4358-8a58-23d4b123be72-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.150961 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.150918 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.151370 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.151326 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20949c65-577f-4bef-9494-b8ddec17c982-original-pull-secret\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.151496 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.151463 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxk9\" (UniqueName: \"kubernetes.io/projected/0693da1f-8341-413f-b174-2575807a5a85-kube-api-access-4lxk9\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.152097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152054 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.152193 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152095 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8g9\" (UniqueName: \"kubernetes.io/projected/d5818964-b07b-4362-9eba-2458e3d58a97-kube-api-access-qn8g9\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.152241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152191 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbvz\" (UniqueName: \"kubernetes.io/projected/f8b282a1-edcf-42d2-9f53-c995efc488cd-kube-api-access-9wbvz\") pod \"kube-state-metrics-5dc8d7d89b-jhh2s\" (UID: \"f8b282a1-edcf-42d2-9f53-c995efc488cd\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jhh2s" Oct 30 05:32:27.152273 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152261 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxfq8\" (UniqueName: \"kubernetes.io/projected/16cdbe91-7a39-4ad3-8ef5-c738a995e26a-kube-api-access-pxfq8\") pod \"network-metrics-daemon-4rx42\" (UID: \"16cdbe91-7a39-4ad3-8ef5-c738a995e26a\") " pod="openshift-multus/network-metrics-daemon-4rx42" Oct 30 05:32:27.152469 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152452 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.152643 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152625 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssgs\" (UniqueName: \"kubernetes.io/projected/794c8751-becf-4dc4-8574-282defa3eb69-kube-api-access-8ssgs\") pod \"network-check-target-phtr7\" (UID: \"794c8751-becf-4dc4-8574-282defa3eb69\") " pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:27.152701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.152626 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29gt\" (UniqueName: \"kubernetes.io/projected/bff6f637-e30a-47b8-88fe-a5aa3dfd5e02-kube-api-access-n29gt\") pod \"monitoring-plugin-77f4f76594-6qhbq\" (UID: \"bff6f637-e30a-47b8-88fe-a5aa3dfd5e02\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:27.153176 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.153159 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hdv\" (UniqueName: \"kubernetes.io/projected/98584416-9e1b-4358-8a58-23d4b123be72-kube-api-access-q4hdv\") pod \"openshift-state-metrics-7bd7489957-wpplk\" (UID: \"98584416-9e1b-4358-8a58-23d4b123be72\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-wpplk" Oct 30 05:32:27.159293 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.159269 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vfz\" (UniqueName: \"kubernetes.io/projected/28230167-b221-4cb3-8601-fe8bd9605a73-kube-api-access-g9vfz\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.159380 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.159337 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdfb\" (UniqueName: \"kubernetes.io/projected/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-kube-api-access-nvdfb\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.159421 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.159410 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cwz\" (UniqueName: \"kubernetes.io/projected/17d8104e-e85c-490a-855d-63af3e241f11-kube-api-access-q4cwz\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.159503 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.159483 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxvws\" (UniqueName: \"kubernetes.io/projected/eddf108a-bf5f-483b-ac39-1804ea6d8d76-kube-api-access-sxvws\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.159684 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.159670 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") pod \"prometheus-k8s-0\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:27.231743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231655 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.231743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231695 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-var-lib-kubelet\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.231743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231711 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-systemd-units\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.231743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231725 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-log-socket\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.231743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231744 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-modprobe-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231769 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-hostroot\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231769 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231805 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-systemd\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231807 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-systemd-units\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231821 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-log-socket\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231810 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-var-lib-kubelet\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231839 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-hostroot\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231834 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-device-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231882 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-kubelet\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231886 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-systemd\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231905 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-kubelet-config\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231922 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-device-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231925 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-modprobe-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231929 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-netns\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231942 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-kubelet\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231962 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-bin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231962 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-kubelet-config\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231972 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-netns\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.231988 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-hosts-file\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232020 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232029 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-bin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232034 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-cnibin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232072 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8885b33d-691c-4ba4-916d-c1e27fbd1cd3-hosts-file\") pod \"node-resolver-q2qpx\" (UID: \"8885b33d-691c-4ba4-916d-c1e27fbd1cd3\") " pod="openshift-dns/node-resolver-q2qpx" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232083 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-cnibin\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232082 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-sys-fs\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232120 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-host\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232138 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-d\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.232097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232142 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-k8s-cni-cncf-io\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232153 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-sys-fs\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232162 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-sys\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232176 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-host\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232193 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232212 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-k8s-cni-cncf-io\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232208 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-sys\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232230 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232229 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-dbus\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232266 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-multus\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232265 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20949c65-577f-4bef-9494-b8ddec17c982-dbus\") pod \"global-pull-secret-syncer-wb5rz\" (UID: \"20949c65-577f-4bef-9494-b8ddec17c982\") " pod="kube-system/global-pull-secret-syncer-wb5rz" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232298 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-system-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232350 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-system-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232376 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-var-lib-cni-multus\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232416 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5818964-b07b-4362-9eba-2458e3d58a97-crio-socket\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232376 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5818964-b07b-4362-9eba-2458e3d58a97-crio-socket\") pod \"insights-runtime-extractor-ctntz\" (UID: \"d5818964-b07b-4362-9eba-2458e3d58a97\") " pod="openshift-insights/insights-runtime-extractor-ctntz" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232440 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-wtmp\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232463 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-lib-modules\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232478 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-socket-dir-parent\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232492 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-netns\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232507 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-etc-kubernetes\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232535 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-system-cni-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232553 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-netns\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232556 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-cnibin\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232572 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-node-exporter-wtmp\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232588 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-cnibin\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232599 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-lib-modules\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232608 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-system-cni-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232621 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-slash\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232610 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-etc-kubernetes\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232611 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-socket-dir-parent\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232593 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-slash\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232661 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-conf-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232675 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232689 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-sys\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232695 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-conf-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232704 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232718 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232734 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-multus-certs\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232749 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232767 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-sys\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232770 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232778 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9766570-5a16-4a4a-aa4e-900bbbed459c-host\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232796 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0693da1f-8341-413f-b174-2575807a5a85-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcx4f\" (UID: \"0693da1f-8341-413f-b174-2575807a5a85\") " pod="openshift-multus/multus-additional-cni-plugins-pcx4f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232816 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-host-run-multus-certs\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232826 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232825 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-etc-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232862 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9766570-5a16-4a4a-aa4e-900bbbed459c-host\") pod \"node-ca-8mh7z\" (UID: \"f9766570-5a16-4a4a-aa4e-900bbbed459c\") " pod="openshift-image-registry/node-ca-8mh7z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232864 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-etc-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232886 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-systemd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232818 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28230167-b221-4cb3-8601-fe8bd9605a73-multus-cni-dir\") pod \"multus-kcw92\" (UID: \"28230167-b221-4cb3-8601-fe8bd9605a73\") " pod="openshift-multus/multus-kcw92" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232911 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-socket-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232931 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-run\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232957 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-conf\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232959 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-systemd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232973 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-ovn\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232983 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-run\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232989 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-socket-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.232993 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysconfig\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233016 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-run-ovn\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233027 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-kubernetes\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233029 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysconfig\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233051 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-var-lib-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233070 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-kubernetes\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233073 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-registration-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233089 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b77f59d5-e2ae-46a9-b3b5-6333ec8ca595-etc-sysctl-conf\") pod \"tuned-fzbh9\" (UID: \"b77f59d5-e2ae-46a9-b3b5-6333ec8ca595\") " pod="openshift-cluster-node-tuning-operator/tuned-fzbh9" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233094 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-var-lib-openvswitch\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233097 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-root\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233150 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/714aff30-d1b2-4ed7-9a37-e7674c4c2ad3-root\") pod \"node-exporter-68zhv\" (UID: \"714aff30-d1b2-4ed7-9a37-e7674c4c2ad3\") " pod="openshift-monitoring/node-exporter-68zhv" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233154 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eddf108a-bf5f-483b-ac39-1804ea6d8d76-host-slash\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233158 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-registration-dir\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233174 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eddf108a-bf5f-483b-ac39-1804ea6d8d76-host-slash\") pod \"iptables-alerter-lp49f\" (UID: \"eddf108a-bf5f-483b-ac39-1804ea6d8d76\") " pod="openshift-network-operator/iptables-alerter-lp49f" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233180 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-kubelet\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233208 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-kubelet\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233209 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-node-log\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233238 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-node-log\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233242 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-bin\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.233220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233266 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-netd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.235190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233289 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-bin\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.235190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233289 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b-host-cni-netd\") pod \"ovnkube-node-cr5xj\" (UID: \"c5c85b4d-dc3f-4677-9fa8-d68410ea9b0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:27.235190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233299 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:27.235190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:27.233355 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17d8104e-e85c-490a-855d-63af3e241f11-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsz9z\" (UID: \"17d8104e-e85c-490a-855d-63af3e241f11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsz9z" Oct 30 05:32:28.059315 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:28.059276 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:32:28.871010 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:28.870980 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxrx6" Oct 30 05:32:29.272387 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:29.272309 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rm8hh" Oct 30 05:32:31.084371 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:31.084343 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:32.050041 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:32.050005 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:34.481382 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:34.481349 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-6qhbq" Oct 30 05:32:36.040042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:36.039966 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:36.040422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:36.040255 5980 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:32:43.085930 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:43.085888 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:32:51.652989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.652953 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:51.653456 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.653382 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="prometheus" containerID="cri-o://a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" gracePeriod=600 Oct 30 05:32:51.653456 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.653389 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="thanos-sidecar" containerID="cri-o://3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" gracePeriod=600 Oct 30 05:32:51.653456 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.653415 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-web" containerID="cri-o://aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" gracePeriod=600 Oct 30 05:32:51.653635 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.653431 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy" containerID="cri-o://5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" gracePeriod=600 Oct 30 05:32:51.653635 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.653444 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="config-reloader" containerID="cri-o://88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" gracePeriod=600 Oct 30 05:32:51.653635 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.653388 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" gracePeriod=600 Oct 30 05:32:51.900045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.900023 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:51.915416 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915358 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915416 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915396 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915419 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915439 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915494 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915512 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915538 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915569 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915608 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915640 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915664 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915697 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915725 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915756 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.915787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915781 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.916200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915815 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.916200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915846 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.916200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915895 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.916200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915945 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.916200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.915959 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.916200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.916036 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") pod \"5abeb78e-0d20-44fe-a201-5855719acd9b\" (UID: \"5abeb78e-0d20-44fe-a201-5855719acd9b\") " Oct 30 05:32:51.916490 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.916345 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.916490 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.916352 5980 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:51.917632 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.917358 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.917632 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.917592 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:51.919076 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.919044 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.919230 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.919098 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.919307 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.919254 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.919370 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.919348 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:51.921893 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.921849 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:51.921893 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.921854 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.922056 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922032 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.922639 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922615 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.922777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922657 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "kube-api-access-87m9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:51.922777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922659 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:51.922777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922697 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.922938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922789 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.922938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.922848 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.923164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.923143 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.923430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.923415 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:51.931632 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:51.931607 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config" pod "5abeb78e-0d20-44fe-a201-5855719acd9b" (UID: "5abeb78e-0d20-44fe-a201-5855719acd9b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:52.017100 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017066 5980 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017100 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017100 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87m9n\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-kube-api-access-87m9n\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017134 5980 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017148 5980 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017161 5980 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017174 5980 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017187 5980 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-config\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017200 5980 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-metrics-client-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017214 5980 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017226 5980 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-config-out\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017238 5980 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-configmap-metrics-client-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017251 5980 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017263 5980 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-db\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017276 5980 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-grpc-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017289 5980 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5abeb78e-0d20-44fe-a201-5855719acd9b-tls-assets\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017302 5980 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-secret-kube-rbac-proxy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017316 5980 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017942 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017330 5980 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5abeb78e-0d20-44fe-a201-5855719acd9b-web-config\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.017942 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.017343 5980 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5abeb78e-0d20-44fe-a201-5855719acd9b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:32:52.156334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156296 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" exitCode=0 Oct 30 05:32:52.156334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156323 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" exitCode=0 Oct 30 05:32:52.156334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156336 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" exitCode=0 Oct 30 05:32:52.156334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156344 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" exitCode=0 Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156352 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" exitCode=0 Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156359 5980 generic.go:356] "Generic (PLEG): container finished" podID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" exitCode=0 Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156374 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156393 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156492 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156535 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156564 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156568 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156602 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} Oct 30 05:32:52.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156639 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} Oct 30 05:32:52.157134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.156678 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5abeb78e-0d20-44fe-a201-5855719acd9b","Type":"ContainerDied","Data":"858c5d8a8e7069f9f35d5cd7a170aab4d63f8abafc845c991997add0223b0a5a"} Oct 30 05:32:52.164585 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.164569 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.171414 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.171398 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.178245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.178225 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.185148 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.185126 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.192609 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.192586 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.199851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.199829 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.206598 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.206569 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.206896 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.206875 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.206957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.206906 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} err="failed to get container status \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" Oct 30 05:32:52.206957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.206939 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.207202 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.207185 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.207251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207210 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} err="failed to get container status \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" Oct 30 05:32:52.207251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207226 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.207463 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.207446 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.207498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207468 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} err="failed to get container status \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" Oct 30 05:32:52.207498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207484 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.207737 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.207717 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.207818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207741 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} err="failed to get container status \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" Oct 30 05:32:52.207818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207756 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.207987 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.207971 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.208023 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.207995 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} err="failed to get container status \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" Oct 30 05:32:52.208023 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208009 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.208264 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.208251 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.208319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208268 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} err="failed to get container status \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" Oct 30 05:32:52.208319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208280 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.208505 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:32:52.208490 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.208539 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208510 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} err="failed to get container status \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" Oct 30 05:32:52.208539 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208523 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.208724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208707 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} err="failed to get container status \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" Oct 30 05:32:52.208768 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208724 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.208932 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208915 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} err="failed to get container status \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" Oct 30 05:32:52.208972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.208932 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.209146 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209129 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} err="failed to get container status \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" Oct 30 05:32:52.209146 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209145 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.209401 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209383 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} err="failed to get container status \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" Oct 30 05:32:52.209459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209402 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.209666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209648 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} err="failed to get container status \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" Oct 30 05:32:52.209666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209664 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.209909 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209885 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} err="failed to get container status \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" Oct 30 05:32:52.209964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.209911 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.210328 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210310 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} err="failed to get container status \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" Oct 30 05:32:52.210328 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210328 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.210554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210536 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} err="failed to get container status \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" Oct 30 05:32:52.210591 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210555 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.210779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210762 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} err="failed to get container status \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" Oct 30 05:32:52.210822 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210780 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.211009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.210988 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} err="failed to get container status \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" Oct 30 05:32:52.211083 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211010 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.211268 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211252 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} err="failed to get container status \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" Oct 30 05:32:52.211316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211268 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.211505 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211478 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} err="failed to get container status \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" Oct 30 05:32:52.211610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211507 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.211982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211874 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} err="failed to get container status \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" Oct 30 05:32:52.211982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.211905 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.212200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212176 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} err="failed to get container status \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" Oct 30 05:32:52.212200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212200 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.212453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212440 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} err="failed to get container status \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" Oct 30 05:32:52.212513 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212454 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.212671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212652 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:52.212749 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212717 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} err="failed to get container status \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" Oct 30 05:32:52.212749 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212739 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.212969 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212950 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} err="failed to get container status \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" Oct 30 05:32:52.213018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.212969 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.213144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213126 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} err="failed to get container status \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" Oct 30 05:32:52.213190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213145 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.213326 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213311 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} err="failed to get container status \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" Oct 30 05:32:52.213366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213326 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.213519 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213503 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} err="failed to get container status \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" Oct 30 05:32:52.213552 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213519 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.213730 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213716 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} err="failed to get container status \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" Oct 30 05:32:52.213770 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213730 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.213934 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213919 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} err="failed to get container status \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" Oct 30 05:32:52.213978 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.213934 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.214165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214144 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} err="failed to get container status \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" Oct 30 05:32:52.214215 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214167 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.214389 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214369 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} err="failed to get container status \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" Oct 30 05:32:52.214430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214390 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.214599 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214579 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} err="failed to get container status \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" Oct 30 05:32:52.214670 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214601 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.214818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214802 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} err="failed to get container status \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" Oct 30 05:32:52.214863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.214817 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.215052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215033 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} err="failed to get container status \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" Oct 30 05:32:52.215133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215052 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.215319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215299 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} err="failed to get container status \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" Oct 30 05:32:52.215361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215321 5980 scope.go:122] "RemoveContainer" containerID="bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089" Oct 30 05:32:52.215530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215516 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089"} err="failed to get container status \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": rpc error: code = NotFound desc = could not find container \"bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089\": container with ID starting with bb0411faabc32fea9dab3b62f730d9a851d3d8d3a1f5a5142d73f52aaac8f089 not found: ID does not exist" Oct 30 05:32:52.215573 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215530 5980 scope.go:122] "RemoveContainer" containerID="5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27" Oct 30 05:32:52.215736 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215719 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27"} err="failed to get container status \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": rpc error: code = NotFound desc = could not find container \"5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27\": container with ID starting with 5da516c99b31d8b1a58dca07e17b5d50c195bf85caf1f3fdbaf7f13df5658f27 not found: ID does not exist" Oct 30 05:32:52.215802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215738 5980 scope.go:122] "RemoveContainer" containerID="aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f" Oct 30 05:32:52.215945 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215929 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f"} err="failed to get container status \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": rpc error: code = NotFound desc = could not find container \"aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f\": container with ID starting with aa414becb55119ef7bab59f96c798fd04449568ee5923b1416ff84aa61740b5f not found: ID does not exist" Oct 30 05:32:52.215989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.215944 5980 scope.go:122] "RemoveContainer" containerID="3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160" Oct 30 05:32:52.216150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216133 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160"} err="failed to get container status \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": rpc error: code = NotFound desc = could not find container \"3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160\": container with ID starting with 3119abd74edb808ba3b969c679316217367757dddce622b2a68cc5d4247b4160 not found: ID does not exist" Oct 30 05:32:52.216189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216151 5980 scope.go:122] "RemoveContainer" containerID="88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0" Oct 30 05:32:52.216355 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216337 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0"} err="failed to get container status \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": rpc error: code = NotFound desc = could not find container \"88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0\": container with ID starting with 88b59e8420ee3f0a7d5c9537989a6df9c891120513476a08390ed8042a8405a0 not found: ID does not exist" Oct 30 05:32:52.216427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216356 5980 scope.go:122] "RemoveContainer" containerID="a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34" Oct 30 05:32:52.216590 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216576 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34"} err="failed to get container status \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": rpc error: code = NotFound desc = could not find container \"a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34\": container with ID starting with a3deaa0e7f92d699f10d4a12b40c74786f72cb4855252a1e381d41301d9f8a34 not found: ID does not exist" Oct 30 05:32:52.216634 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216589 5980 scope.go:122] "RemoveContainer" containerID="926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb" Oct 30 05:32:52.216756 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.216742 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb"} err="failed to get container status \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": rpc error: code = NotFound desc = could not find container \"926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb\": container with ID starting with 926c5bb8c98cacc0167a8e525525c5a93bf32a4f67751ae4fba4d67596606ccb not found: ID does not exist" Oct 30 05:32:52.223948 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.223918 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:52.253968 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.253940 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:52.254200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254187 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="config-reloader" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254203 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="config-reloader" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254215 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="prometheus" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254220 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="prometheus" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254230 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-web" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254236 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-web" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254245 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy" Oct 30 05:32:52.254251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254250 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254255 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="init-config-reloader" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254260 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="init-config-reloader" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254268 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="thanos-sidecar" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254273 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="thanos-sidecar" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254282 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-thanos" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254286 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-thanos" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254328 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="config-reloader" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254334 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="prometheus" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254341 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-web" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254346 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254351 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="init-config-reloader" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254357 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="thanos-sidecar" Oct 30 05:32:52.254446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.254362 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" containerName="kube-rbac-proxy-thanos" Oct 30 05:32:52.257872 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.257858 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.274206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274178 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:32:52.274370 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274205 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:32:52.274522 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274502 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:32:52.274582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274520 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:32:52.274582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274545 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:32:52.274582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274551 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:32:52.274582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274547 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:32:52.274582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274557 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:32:52.274743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274585 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:32:52.274743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274600 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7mkgv\"" Oct 30 05:32:52.274743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.274682 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2eaov2pk6s7tl\"" Oct 30 05:32:52.281060 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.281043 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:32:52.281060 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.281056 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:32:52.281329 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.281101 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:32:52.281491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.281346 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:32:52.294997 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.294972 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:32:52.294997 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.294983 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:32:52.295596 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.295580 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:52.319854 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.319818 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.319854 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.319856 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.319880 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.319937 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320025 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-config\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320044 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwz9\" (UniqueName: \"kubernetes.io/projected/10ea2de1-bf89-40da-a84b-2350d913bd66-kube-api-access-lgwz9\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320061 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320082 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320125 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320156 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320194 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10ea2de1-bf89-40da-a84b-2350d913bd66-config-out\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320253 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320277 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-web-config\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320297 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10ea2de1-bf89-40da-a84b-2350d913bd66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320316 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320336 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320384 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320423 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320448 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.320491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.320464 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421232 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421196 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421232 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421236 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421254 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421375 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421408 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10ea2de1-bf89-40da-a84b-2350d913bd66-config-out\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421457 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421482 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-web-config\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421516 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10ea2de1-bf89-40da-a84b-2350d913bd66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421554 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421589 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421622 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421660 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421688 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421715 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.421764 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421745 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421772 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421796 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421820 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421849 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-config\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421871 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwz9\" (UniqueName: \"kubernetes.io/projected/10ea2de1-bf89-40da-a84b-2350d913bd66-kube-api-access-lgwz9\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421874 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421655 5980 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:32:52.422162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.421953 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422511 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.422416 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.422866 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.422817 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.423010 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.422983 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.424949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.424922 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/10ea2de1-bf89-40da-a84b-2350d913bd66-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.425268 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.425143 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.425268 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.425227 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.425617 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.425292 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/10ea2de1-bf89-40da-a84b-2350d913bd66-config-out\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.425617 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.425316 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.425617 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.425549 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.425791 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.425772 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.426346 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.426321 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-config\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.426457 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.426436 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.426699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.426604 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/10ea2de1-bf89-40da-a84b-2350d913bd66-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.427237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.427214 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-web-config\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.427680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.427658 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.428093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.428076 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.428378 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.428357 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/10ea2de1-bf89-40da-a84b-2350d913bd66-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.449354 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.449335 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwz9\" (UniqueName: \"kubernetes.io/projected/10ea2de1-bf89-40da-a84b-2350d913bd66-kube-api-access-lgwz9\") pod \"prometheus-k8s-0\" (UID: \"10ea2de1-bf89-40da-a84b-2350d913bd66\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.574189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.574149 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:52.909313 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:52.909275 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:32:52.912075 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:32:52.912037 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ea2de1_bf89_40da_a84b_2350d913bd66.slice/crio-41ea2286850c869ec2fb768270dd719140bc771dd6ce0c72367a408d39373b3a WatchSource:0}: Error finding container 41ea2286850c869ec2fb768270dd719140bc771dd6ce0c72367a408d39373b3a: Status 404 returned error can't find the container with id 41ea2286850c869ec2fb768270dd719140bc771dd6ce0c72367a408d39373b3a Oct 30 05:32:53.161725 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:53.161694 5980 generic.go:356] "Generic (PLEG): container finished" podID="10ea2de1-bf89-40da-a84b-2350d913bd66" containerID="fd73f13f5043ef333692b4a55b6f9c8ec6da87c21694bd6d04866cfebb3f430d" exitCode=0 Oct 30 05:32:53.161935 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:53.161790 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerDied","Data":"fd73f13f5043ef333692b4a55b6f9c8ec6da87c21694bd6d04866cfebb3f430d"} Oct 30 05:32:53.161935 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:53.161870 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"41ea2286850c869ec2fb768270dd719140bc771dd6ce0c72367a408d39373b3a"} Oct 30 05:32:54.010634 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.010601 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abeb78e-0d20-44fe-a201-5855719acd9b" path="/var/lib/kubelet/pods/5abeb78e-0d20-44fe-a201-5855719acd9b/volumes" Oct 30 05:32:54.169355 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.169322 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"9dc7b7ebbd422f7fdbccd76f4d1e8a64ea6dde11c98fd330ca489ddc619c3a6a"} Oct 30 05:32:54.169521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.169385 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"1b7a9c224947cde75e7add752552edf9ae81efcbc1f3ddf83437c2578d4f7971"} Oct 30 05:32:54.169521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.169414 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"b638eb0121370ba98225c7a31ab6c63c405bc4058a46d5c742a1ce66d116fe08"} Oct 30 05:32:54.169521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.169476 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"13d7eed1b8a7cdd24d8f6bc24cc97665109ba7ec022b502d44b9308e1c9f1fa9"} Oct 30 05:32:54.169521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.169500 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"847534b5bde415a1f357cfa07292eea79ac22fb491e18179e817c3afeac44593"} Oct 30 05:32:54.169732 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.169527 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"10ea2de1-bf89-40da-a84b-2350d913bd66","Type":"ContainerStarted","Data":"7baa641928aa5b4ff019cca202e7ef2b51a89f94552f62f8b7008998af6c2733"} Oct 30 05:32:54.241099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:54.241045 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.222867239 podStartE2EDuration="2.241032409s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.16517ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:54.240727424 +0000 UTC m=+28.348766456" watchObservedRunningTime="2026-10-30 05:32:54.241032409 +0000 UTC m=+28.349071476" Oct 30 05:32:56.540031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:56.539991 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cr5xj" Oct 30 05:32:57.574780 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:57.574744 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:58.071165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:32:58.071132 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-phtr7" Oct 30 05:33:13.086314 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:33:13.086278 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:33:26.531431 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:33:26.531392 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:33:26.547561 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:33:26.547533 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:33:27.297535 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:33:27.297509 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:33:43.087054 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:33:43.087015 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:34:13.085259 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:34:13.085181 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:34:43.087382 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:34:43.087349 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:35:13.085758 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:35:13.085710 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:35:43.087233 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:35:43.087152 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:36:13.086883 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:36:13.086847 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:36:43.087020 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:36:43.086973 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:36:51.904943 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.904911 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-fl76c"] Oct 30 05:36:51.907165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.907144 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:51.934358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.934321 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 05:36:51.934358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.934334 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 05:36:51.934582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.934397 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f5fbb\"" Oct 30 05:36:51.934844 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.934823 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fl76c"] Oct 30 05:36:51.999026 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.998992 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68a42cf6-cf09-4482-a275-25087e5cb06d-bound-sa-token\") pod \"cert-manager-597b95cb84-fl76c\" (UID: \"68a42cf6-cf09-4482-a275-25087e5cb06d\") " pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:51.999026 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:51.999023 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tltkh\" (UniqueName: \"kubernetes.io/projected/68a42cf6-cf09-4482-a275-25087e5cb06d-kube-api-access-tltkh\") pod \"cert-manager-597b95cb84-fl76c\" (UID: \"68a42cf6-cf09-4482-a275-25087e5cb06d\") " pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:52.099711 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.099669 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68a42cf6-cf09-4482-a275-25087e5cb06d-bound-sa-token\") pod \"cert-manager-597b95cb84-fl76c\" (UID: \"68a42cf6-cf09-4482-a275-25087e5cb06d\") " pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:52.099711 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.099705 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tltkh\" (UniqueName: \"kubernetes.io/projected/68a42cf6-cf09-4482-a275-25087e5cb06d-kube-api-access-tltkh\") pod \"cert-manager-597b95cb84-fl76c\" (UID: \"68a42cf6-cf09-4482-a275-25087e5cb06d\") " pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:52.123522 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.123482 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68a42cf6-cf09-4482-a275-25087e5cb06d-bound-sa-token\") pod \"cert-manager-597b95cb84-fl76c\" (UID: \"68a42cf6-cf09-4482-a275-25087e5cb06d\") " pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:52.123647 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.123580 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tltkh\" (UniqueName: \"kubernetes.io/projected/68a42cf6-cf09-4482-a275-25087e5cb06d-kube-api-access-tltkh\") pod \"cert-manager-597b95cb84-fl76c\" (UID: \"68a42cf6-cf09-4482-a275-25087e5cb06d\") " pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:52.224204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.224099 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fl76c" Oct 30 05:36:52.518752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.518664 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fl76c"] Oct 30 05:36:52.522400 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:36:52.522372 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68a42cf6_cf09_4482_a275_25087e5cb06d.slice/crio-5394f600333ed81a6d9b33ea0f0a4728b63b7476261d21cdbad06f3e02659990 WatchSource:0}: Error finding container 5394f600333ed81a6d9b33ea0f0a4728b63b7476261d21cdbad06f3e02659990: Status 404 returned error can't find the container with id 5394f600333ed81a6d9b33ea0f0a4728b63b7476261d21cdbad06f3e02659990 Oct 30 05:36:52.524360 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.524340 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:36:52.957165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:52.957126 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fl76c" event={"ID":"68a42cf6-cf09-4482-a275-25087e5cb06d","Type":"ContainerStarted","Data":"5394f600333ed81a6d9b33ea0f0a4728b63b7476261d21cdbad06f3e02659990"} Oct 30 05:36:55.971042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:55.971001 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fl76c" event={"ID":"68a42cf6-cf09-4482-a275-25087e5cb06d","Type":"ContainerStarted","Data":"4fbcf47a34e3346d031d8f45e4e6a77f2e7cff79f7bdcbda376eaca33a51f4cf"} Oct 30 05:36:56.012406 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:36:56.012299 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-fl76c" podStartSLOduration=1.786250877 podStartE2EDuration="5.01228186s" totalImagesPullingTime="3.226030983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:56.011489908 +0000 UTC m=+270.119528940" watchObservedRunningTime="2026-10-30 05:36:56.01228186 +0000 UTC m=+270.120320893" Oct 30 05:37:13.086582 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:37:13.086542 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:37:17.922853 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:17.922820 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp"] Oct 30 05:37:17.926372 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:17.926348 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:17.962085 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:17.962057 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp"] Oct 30 05:37:18.004498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.004443 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.004498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.004491 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9253b348-62ae-4bf7-b6fe-8b25d841ec93-tmp\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.004752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.004578 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9253b348-62ae-4bf7-b6fe-8b25d841ec93-manifests\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.004752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.004642 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.004818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.004773 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.004851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.004820 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf5rt\" (UniqueName: \"kubernetes.io/projected/9253b348-62ae-4bf7-b6fe-8b25d841ec93-kube-api-access-xf5rt\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.006958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.006934 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 05:37:18.007097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.006983 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 05:37:18.007097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.007025 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 05:37:18.007239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.007194 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 05:37:18.007239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.007196 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dmsd5\"" Oct 30 05:37:18.007239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.007197 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 05:37:18.105462 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.105416 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.105462 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.105465 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf5rt\" (UniqueName: \"kubernetes.io/projected/9253b348-62ae-4bf7-b6fe-8b25d841ec93-kube-api-access-xf5rt\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.105745 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.105523 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.105745 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.105549 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9253b348-62ae-4bf7-b6fe-8b25d841ec93-tmp\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.105745 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.105579 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9253b348-62ae-4bf7-b6fe-8b25d841ec93-manifests\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.105745 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.105618 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.106098 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.106055 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9253b348-62ae-4bf7-b6fe-8b25d841ec93-manifests\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.108131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.108087 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9253b348-62ae-4bf7-b6fe-8b25d841ec93-tmp\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.108336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.108317 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.108471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.108452 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.108532 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.108454 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9253b348-62ae-4bf7-b6fe-8b25d841ec93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.154027 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.153994 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf5rt\" (UniqueName: \"kubernetes.io/projected/9253b348-62ae-4bf7-b6fe-8b25d841ec93-kube-api-access-xf5rt\") pod \"opendatahub-operator-controller-manager-f7d897f7-g8rjp\" (UID: \"9253b348-62ae-4bf7-b6fe-8b25d841ec93\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.240827 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.240736 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:18.604495 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:18.604458 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp"] Oct 30 05:37:18.611848 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:37:18.611804 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9253b348_62ae_4bf7_b6fe_8b25d841ec93.slice/crio-00832c30a2fcf15399a0a8d40ed497774149cb8f79531786161a87d09a3908ba WatchSource:0}: Error finding container 00832c30a2fcf15399a0a8d40ed497774149cb8f79531786161a87d09a3908ba: Status 404 returned error can't find the container with id 00832c30a2fcf15399a0a8d40ed497774149cb8f79531786161a87d09a3908ba Oct 30 05:37:19.052130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:19.052022 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" event={"ID":"9253b348-62ae-4bf7-b6fe-8b25d841ec93","Type":"ContainerStarted","Data":"00832c30a2fcf15399a0a8d40ed497774149cb8f79531786161a87d09a3908ba"} Oct 30 05:37:22.064933 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:22.064897 5980 generic.go:356] "Generic (PLEG): container finished" podID="9253b348-62ae-4bf7-b6fe-8b25d841ec93" containerID="6b67bc5ffeb7ca5276bb75b42ad0ff52e657f536c0936f12e6d5d60901580701" exitCode=0 Oct 30 05:37:22.065345 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:22.064974 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" event={"ID":"9253b348-62ae-4bf7-b6fe-8b25d841ec93","Type":"ContainerDied","Data":"6b67bc5ffeb7ca5276bb75b42ad0ff52e657f536c0936f12e6d5d60901580701"} Oct 30 05:37:23.071560 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:23.071523 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" event={"ID":"9253b348-62ae-4bf7-b6fe-8b25d841ec93","Type":"ContainerStarted","Data":"fd95a57fe85c8034476e8a624e967959ebbc1b851f3d810902c5104610a2c554"} Oct 30 05:37:23.072042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:23.071661 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:23.159219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:23.159164 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" podStartSLOduration=2.671804648 podStartE2EDuration="6.159149405s" totalImagesPullingTime="3.074073807s" totalInitContainerRuntime="413.27095ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:23.146344983 +0000 UTC m=+297.254384040" watchObservedRunningTime="2026-10-30 05:37:23.159149405 +0000 UTC m=+297.267188438" Oct 30 05:37:34.077520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:34.077458 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-g8rjp" Oct 30 05:37:35.472019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.471986 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl"] Oct 30 05:37:35.477975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.477955 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.503289 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.503256 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 05:37:35.508880 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.508857 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 05:37:35.516344 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.516318 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl"] Oct 30 05:37:35.597580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.597554 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:37:35.597762 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.597654 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 05:37:35.604471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.604450 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d8qd8\"" Oct 30 05:37:35.604471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.604467 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 05:37:35.656851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.656821 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/444497f5-834b-43e2-97b9-c93b687d211a-manager-config\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.657231 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.656863 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/444497f5-834b-43e2-97b9-c93b687d211a-metrics-cert\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.657231 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.656963 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444497f5-834b-43e2-97b9-c93b687d211a-cert\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.657231 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.657009 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjpvp\" (UniqueName: \"kubernetes.io/projected/444497f5-834b-43e2-97b9-c93b687d211a-kube-api-access-zjpvp\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.758285 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.758193 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/444497f5-834b-43e2-97b9-c93b687d211a-manager-config\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.758285 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.758237 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/444497f5-834b-43e2-97b9-c93b687d211a-metrics-cert\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.758285 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.758284 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444497f5-834b-43e2-97b9-c93b687d211a-cert\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.758519 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.758322 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjpvp\" (UniqueName: \"kubernetes.io/projected/444497f5-834b-43e2-97b9-c93b687d211a-kube-api-access-zjpvp\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.758982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.758954 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/444497f5-834b-43e2-97b9-c93b687d211a-manager-config\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.760862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.760835 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/444497f5-834b-43e2-97b9-c93b687d211a-metrics-cert\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.760967 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.760933 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/444497f5-834b-43e2-97b9-c93b687d211a-cert\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.782592 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.782566 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjpvp\" (UniqueName: \"kubernetes.io/projected/444497f5-834b-43e2-97b9-c93b687d211a-kube-api-access-zjpvp\") pod \"lws-controller-manager-5d64f7d9-j2mwl\" (UID: \"444497f5-834b-43e2-97b9-c93b687d211a\") " pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:35.788445 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:35.788423 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:36.093680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:36.093646 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl"] Oct 30 05:37:36.096866 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:37:36.096837 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444497f5_834b_43e2_97b9_c93b687d211a.slice/crio-70275f238dea1b59b25f1b03836b88c04d7e7dd750a2df203184edf17165aa2c WatchSource:0}: Error finding container 70275f238dea1b59b25f1b03836b88c04d7e7dd750a2df203184edf17165aa2c: Status 404 returned error can't find the container with id 70275f238dea1b59b25f1b03836b88c04d7e7dd750a2df203184edf17165aa2c Oct 30 05:37:36.117429 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:36.117393 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" event={"ID":"444497f5-834b-43e2-97b9-c93b687d211a","Type":"ContainerStarted","Data":"70275f238dea1b59b25f1b03836b88c04d7e7dd750a2df203184edf17165aa2c"} Oct 30 05:37:37.775152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.774186 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd"] Oct 30 05:37:37.777875 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.777844 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.817914 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.817856 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 05:37:37.817914 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.817900 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 05:37:37.831163 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.830521 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p9n4d\"" Oct 30 05:37:37.831370 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.831300 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd"] Oct 30 05:37:37.854441 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.854401 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:37:37.854441 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.854402 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:37:37.877080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.877042 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6de8-58d0-4fba-9193-a23327d0d792-tmp\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.877277 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.877131 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jczr\" (UniqueName: \"kubernetes.io/projected/ef9b6de8-58d0-4fba-9193-a23327d0d792-kube-api-access-8jczr\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.877277 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.877177 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9b6de8-58d0-4fba-9193-a23327d0d792-tls-certs\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.977605 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.977568 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9b6de8-58d0-4fba-9193-a23327d0d792-tls-certs\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.977779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.977616 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6de8-58d0-4fba-9193-a23327d0d792-tmp\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.977779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.977665 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jczr\" (UniqueName: \"kubernetes.io/projected/ef9b6de8-58d0-4fba-9193-a23327d0d792-kube-api-access-8jczr\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.979982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.979953 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef9b6de8-58d0-4fba-9193-a23327d0d792-tmp\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:37.980464 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:37.980444 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef9b6de8-58d0-4fba-9193-a23327d0d792-tls-certs\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:38.004180 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:38.004151 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jczr\" (UniqueName: \"kubernetes.io/projected/ef9b6de8-58d0-4fba-9193-a23327d0d792-kube-api-access-8jczr\") pod \"kube-auth-proxy-8596b47b87-pk7dd\" (UID: \"ef9b6de8-58d0-4fba-9193-a23327d0d792\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:38.094537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:38.094504 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" Oct 30 05:37:38.406254 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:38.406182 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd"] Oct 30 05:37:38.409705 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:37:38.409676 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9b6de8_58d0_4fba_9193_a23327d0d792.slice/crio-b404971e7c5269c545d6d5bf8bb98c750b86d0a147442133626f394814fa6b99 WatchSource:0}: Error finding container b404971e7c5269c545d6d5bf8bb98c750b86d0a147442133626f394814fa6b99: Status 404 returned error can't find the container with id b404971e7c5269c545d6d5bf8bb98c750b86d0a147442133626f394814fa6b99 Oct 30 05:37:39.130888 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:39.130853 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" event={"ID":"ef9b6de8-58d0-4fba-9193-a23327d0d792","Type":"ContainerStarted","Data":"b404971e7c5269c545d6d5bf8bb98c750b86d0a147442133626f394814fa6b99"} Oct 30 05:37:40.140124 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:40.140080 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" event={"ID":"444497f5-834b-43e2-97b9-c93b687d211a","Type":"ContainerStarted","Data":"bb390fdc5be01c6ec419fff45f72a870d26f103290c99f96eb9c630037d877fa"} Oct 30 05:37:40.140608 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:40.140193 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:37:40.188645 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:40.188578 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" podStartSLOduration=2.129915929 podStartE2EDuration="5.188562398s" totalImagesPullingTime="3.058646469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:40.188019402 +0000 UTC m=+314.296058435" watchObservedRunningTime="2026-10-30 05:37:40.188562398 +0000 UTC m=+314.296601483" Oct 30 05:37:41.145355 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:41.145269 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" event={"ID":"ef9b6de8-58d0-4fba-9193-a23327d0d792","Type":"ContainerStarted","Data":"5320196dd17ece9f985bd652604921da4788c78442220a667d4cb45b63bd0bb0"} Oct 30 05:37:41.188734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:41.188681 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8596b47b87-pk7dd" podStartSLOduration=1.7155596339999999 podStartE2EDuration="4.188667161s" totalImagesPullingTime="2.473107527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:41.188321855 +0000 UTC m=+315.296360889" watchObservedRunningTime="2026-10-30 05:37:41.188667161 +0000 UTC m=+315.296706194" Oct 30 05:37:43.086306 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:37:43.086266 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:37:51.148031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:37:51.147994 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d64f7d9-j2mwl" Oct 30 05:38:13.088271 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:38:13.088233 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:38:43.086159 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:38:43.086052 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:39:13.086058 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:39:13.085964 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:39:43.089265 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:39:43.089220 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:40:10.485031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.484996 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z"] Oct 30 05:40:10.488287 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.488272 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.501909 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.501884 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z"] Oct 30 05:40:10.509713 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.509685 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n9wqs\"" Oct 30 05:40:10.514760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.514736 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 05:40:10.514760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.514738 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 05:40:10.581702 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.581661 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tmd\" (UniqueName: \"kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd\") pod \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.581918 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.581718 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.683276 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.683242 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tmd\" (UniqueName: \"kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd\") pod \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.683515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.683290 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.683748 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.683723 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.708753 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.708718 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tmd\" (UniqueName: \"kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd\") pod \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:10.802184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:10.802085 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:11.132653 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:11.132617 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z"] Oct 30 05:40:11.136047 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:40:11.136015 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edefd0a_3b2a_48c7_b230_b5604a8a9d9c.slice/crio-74cd48614b0d05517c312000c6b2300a97c5944dade2576378922f444cfce342 WatchSource:0}: Error finding container 74cd48614b0d05517c312000c6b2300a97c5944dade2576378922f444cfce342: Status 404 returned error can't find the container with id 74cd48614b0d05517c312000c6b2300a97c5944dade2576378922f444cfce342 Oct 30 05:40:11.682855 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:11.682810 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" event={"ID":"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c","Type":"ContainerStarted","Data":"74cd48614b0d05517c312000c6b2300a97c5944dade2576378922f444cfce342"} Oct 30 05:40:13.091047 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:13.091002 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:40:15.705419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:15.705377 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" event={"ID":"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c","Type":"ContainerStarted","Data":"0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af"} Oct 30 05:40:15.705871 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:15.705501 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:15.783964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:15.783910 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" podStartSLOduration=2.156141146 podStartE2EDuration="5.783896527s" totalImagesPullingTime="3.627755381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:15.783660413 +0000 UTC m=+469.891699445" watchObservedRunningTime="2026-10-30 05:40:15.783896527 +0000 UTC m=+469.891935559" Oct 30 05:40:18.228168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.228132 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8"] Oct 30 05:40:18.231474 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.231454 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.250709 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.250674 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27863d34-3115-428c-a857-077f149b18ae-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.250864 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.250731 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27863d34-3115-428c-a857-077f149b18ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.250864 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.250796 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2js\" (UniqueName: \"kubernetes.io/projected/27863d34-3115-428c-a857-077f149b18ae-kube-api-access-jv2js\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.250986 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.250962 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8"] Oct 30 05:40:18.252921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.252896 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 05:40:18.263571 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.263549 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 05:40:18.263692 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.263637 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8fkww\"" Oct 30 05:40:18.351610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.351569 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27863d34-3115-428c-a857-077f149b18ae-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.351610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.351616 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27863d34-3115-428c-a857-077f149b18ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.351839 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.351665 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2js\" (UniqueName: \"kubernetes.io/projected/27863d34-3115-428c-a857-077f149b18ae-kube-api-access-jv2js\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.351839 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:18.351780 5980 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Oct 30 05:40:18.351921 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:18.351854 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27863d34-3115-428c-a857-077f149b18ae-plugin-serving-cert podName:27863d34-3115-428c-a857-077f149b18ae nodeName:}" failed. No retries permitted until 2026-10-30 05:40:18.851836291 +0000 UTC m=+472.959875316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/27863d34-3115-428c-a857-077f149b18ae-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-vfcq8" (UID: "27863d34-3115-428c-a857-077f149b18ae") : secret "plugin-serving-cert" not found Oct 30 05:40:18.352235 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.352217 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/27863d34-3115-428c-a857-077f149b18ae-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.376924 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.376894 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2js\" (UniqueName: \"kubernetes.io/projected/27863d34-3115-428c-a857-077f149b18ae-kube-api-access-jv2js\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.856890 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.856854 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27863d34-3115-428c-a857-077f149b18ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:18.859398 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:18.859374 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/27863d34-3115-428c-a857-077f149b18ae-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vfcq8\" (UID: \"27863d34-3115-428c-a857-077f149b18ae\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:19.142302 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:19.142210 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" Oct 30 05:40:19.438096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:19.437984 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8"] Oct 30 05:40:19.441520 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:40:19.441493 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27863d34_3115_428c_a857_077f149b18ae.slice/crio-fa3e13e9d5e46d7a0250c5c53f84de1a0fea83a42781ed305b8e61d1b8f2aa82 WatchSource:0}: Error finding container fa3e13e9d5e46d7a0250c5c53f84de1a0fea83a42781ed305b8e61d1b8f2aa82: Status 404 returned error can't find the container with id fa3e13e9d5e46d7a0250c5c53f84de1a0fea83a42781ed305b8e61d1b8f2aa82 Oct 30 05:40:19.721965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:19.721877 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" event={"ID":"27863d34-3115-428c-a857-077f149b18ae","Type":"ContainerStarted","Data":"fa3e13e9d5e46d7a0250c5c53f84de1a0fea83a42781ed305b8e61d1b8f2aa82"} Oct 30 05:40:24.746041 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:24.746004 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" event={"ID":"27863d34-3115-428c-a857-077f149b18ae","Type":"ContainerStarted","Data":"fe5e9ae665c84fe17c7861ba465b63ff653fa79f2ba8ca268a5aa33ccccdab06"} Oct 30 05:40:24.791777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:24.791715 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vfcq8" podStartSLOduration=2.26708938 podStartE2EDuration="6.79169816s" totalImagesPullingTime="4.52460878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:24.790291124 +0000 UTC m=+478.898330156" watchObservedRunningTime="2026-10-30 05:40:24.79169816 +0000 UTC m=+478.899737193" Oct 30 05:40:26.711659 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:26.711627 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:28.039868 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.039835 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg"] Oct 30 05:40:28.044488 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.044466 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.075164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.075136 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg"] Oct 30 05:40:28.141163 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.141128 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.141317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.141179 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2mb\" (UniqueName: \"kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb\") pod \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.242075 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.242036 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.242280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.242087 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2mb\" (UniqueName: \"kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb\") pod \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.242484 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.242463 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.266751 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.266722 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2mb\" (UniqueName: \"kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb\") pod \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.356932 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.356897 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:28.693813 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.693789 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg"] Oct 30 05:40:28.695726 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:40:28.695696 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b421985_9d2b_474e_870d_20e1a6d416b1.slice/crio-d022bb2f5bd7ab9da956036b5687c868628cf4cd9bf5b4a29d41fa2b4e6eacc9 WatchSource:0}: Error finding container d022bb2f5bd7ab9da956036b5687c868628cf4cd9bf5b4a29d41fa2b4e6eacc9: Status 404 returned error can't find the container with id d022bb2f5bd7ab9da956036b5687c868628cf4cd9bf5b4a29d41fa2b4e6eacc9 Oct 30 05:40:28.763990 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:28.763952 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" event={"ID":"0b421985-9d2b-474e-870d-20e1a6d416b1","Type":"ContainerStarted","Data":"d022bb2f5bd7ab9da956036b5687c868628cf4cd9bf5b4a29d41fa2b4e6eacc9"} Oct 30 05:40:29.770068 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:29.770025 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" event={"ID":"0b421985-9d2b-474e-870d-20e1a6d416b1","Type":"ContainerStarted","Data":"963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47"} Oct 30 05:40:29.770529 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:29.770118 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:29.846031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:29.845978 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" podStartSLOduration=1.845964154 podStartE2EDuration="1.845964154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:29.845411747 +0000 UTC m=+483.953450846" watchObservedRunningTime="2026-10-30 05:40:29.845964154 +0000 UTC m=+483.954003187" Oct 30 05:40:30.400416 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.400364 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z"] Oct 30 05:40:30.400641 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.400607 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" containerName="manager" containerID="cri-o://0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af" gracePeriod=2 Oct 30 05:40:30.401063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.401041 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z"] Oct 30 05:40:30.413884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.413851 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm"] Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.416127 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.416501 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" containerName="manager" Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.416527 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" containerName="manager" Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.416746 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" containerName="manager" Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.422313 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.426450 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg"] Oct 30 05:40:30.431065 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.427607 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg"] Oct 30 05:40:30.437705 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.437671 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.439625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.439573 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm"] Oct 30 05:40:30.447468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.447444 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6"] Oct 30 05:40:30.448031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.448009 5980 cpu_manager.go:414] "containerMap: removing container" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" containerName="manager" Oct 30 05:40:30.448159 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.448034 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" containerName="manager" Oct 30 05:40:30.448159 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.448152 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" containerName="manager" Oct 30 05:40:30.451358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.451286 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.462672 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.462412 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8rb\" (UniqueName: \"kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb\") pod \"kuadrant-operator-controller-manager-6b6fb56846-9nbsm\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.462672 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.462490 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-9nbsm\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.487622 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.487586 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6"] Oct 30 05:40:30.508272 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.508230 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.563859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.563826 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-9nbsm\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.564027 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.563884 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb9wk\" (UniqueName: \"kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk\") pod \"kuadrant-operator-controller-manager-6b76784d48-crtp6\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.564027 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.563913 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-crtp6\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.564027 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.564005 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8rb\" (UniqueName: \"kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb\") pod \"kuadrant-operator-controller-manager-6b6fb56846-9nbsm\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.564242 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.564223 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-9nbsm\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.592303 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.592280 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8rb\" (UniqueName: \"kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb\") pod \"kuadrant-operator-controller-manager-6b6fb56846-9nbsm\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.651877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.651821 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.664871 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.664848 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb9wk\" (UniqueName: \"kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk\") pod \"kuadrant-operator-controller-manager-6b76784d48-crtp6\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.664956 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.664887 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-crtp6\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.665260 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.665244 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-crtp6\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.666759 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.666735 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.689863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.689832 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb9wk\" (UniqueName: \"kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk\") pod \"kuadrant-operator-controller-manager-6b76784d48-crtp6\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.765955 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.765918 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd\" (UniqueName: \"kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd\") pod \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " Oct 30 05:40:30.766151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.765986 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume\") pod \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\" (UID: \"9edefd0a-3b2a-48c7-b230-b5604a8a9d9c\") " Oct 30 05:40:30.766406 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.766379 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume" pod "9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" (UID: "9edefd0a-3b2a-48c7-b230-b5604a8a9d9c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:30.768342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.768319 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd" pod "9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" (UID: "9edefd0a-3b2a-48c7-b230-b5604a8a9d9c"). InnerVolumeSpecName "kube-api-access-84tmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:30.775542 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.775513 5980 generic.go:356] "Generic (PLEG): container finished" podID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" containerID="0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af" exitCode=0 Oct 30 05:40:30.775938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.775569 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.775938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.775614 5980 scope.go:122] "RemoveContainer" containerID="0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af" Oct 30 05:40:30.783940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.783922 5980 scope.go:122] "RemoveContainer" containerID="0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af" Oct 30 05:40:30.784202 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.784182 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af\": container with ID starting with 0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af not found: ID does not exist" containerID="0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af" Oct 30 05:40:30.784260 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.784211 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af"} err="failed to get container status \"0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af\": rpc error: code = NotFound desc = could not find container \"0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af\": container with ID starting with 0e54a784565c936e26ffe3d745db3a9260c5289573d35db7832950352fc1c8af not found: ID does not exist" Oct 30 05:40:30.790638 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.790610 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.801845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.801825 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:30.805445 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:30.805420 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:30.808048 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.808025 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:30.867735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.867697 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84tmd\" (UniqueName: \"kubernetes.io/projected/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-kube-api-access-84tmd\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:30.867735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:30.867736 5980 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c-extensions-socket-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:31.133440 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.133171 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm"] Oct 30 05:40:31.133947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.133774 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6"] Oct 30 05:40:31.136137 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:40:31.136097 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81b740b_aedd_428a_9b68_2b91238165e2.slice/crio-7cfaec7a1c647ce641a1ce0cab9e03658fd0ecac0b66431850b5424625c49fe8 WatchSource:0}: Error finding container 7cfaec7a1c647ce641a1ce0cab9e03658fd0ecac0b66431850b5424625c49fe8: Status 404 returned error can't find the container with id 7cfaec7a1c647ce641a1ce0cab9e03658fd0ecac0b66431850b5424625c49fe8 Oct 30 05:40:31.136904 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:40:31.136883 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b33cf5_b1f6_44c0_93c6_f7ae6e6dc4c1.slice/crio-87c7762c9ba27d5f233d98b594f32fad576f0130fd3ae583e8f07456fada4158 WatchSource:0}: Error finding container 87c7762c9ba27d5f233d98b594f32fad576f0130fd3ae583e8f07456fada4158: Status 404 returned error can't find the container with id 87c7762c9ba27d5f233d98b594f32fad576f0130fd3ae583e8f07456fada4158 Oct 30 05:40:31.781624 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.781536 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" event={"ID":"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1","Type":"ContainerStarted","Data":"d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7"} Oct 30 05:40:31.781624 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.781592 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" event={"ID":"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1","Type":"ContainerStarted","Data":"87c7762c9ba27d5f233d98b594f32fad576f0130fd3ae583e8f07456fada4158"} Oct 30 05:40:31.781624 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.781620 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:31.784172 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.784147 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" event={"ID":"e81b740b-aedd-428a-9b68-2b91238165e2","Type":"ContainerStarted","Data":"6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90"} Oct 30 05:40:31.784306 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.784179 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" containerName="manager" containerID="cri-o://963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47" gracePeriod=2 Oct 30 05:40:31.784306 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.784185 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" event={"ID":"e81b740b-aedd-428a-9b68-2b91238165e2","Type":"ContainerStarted","Data":"7cfaec7a1c647ce641a1ce0cab9e03658fd0ecac0b66431850b5424625c49fe8"} Oct 30 05:40:31.784422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.784316 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:31.827739 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.827697 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" podStartSLOduration=1.827682303 podStartE2EDuration="1.827682303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:31.827428474 +0000 UTC m=+485.935467508" watchObservedRunningTime="2026-10-30 05:40:31.827682303 +0000 UTC m=+485.935721336" Oct 30 05:40:31.842081 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:31.842049 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:31.856860 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:31.856830 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:31.871598 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:31.871565 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-dxc7z\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-dxc7z" Oct 30 05:40:31.922577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:31.922518 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" podStartSLOduration=1.922498856 podStartE2EDuration="1.922498856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:31.921406196 +0000 UTC m=+486.029445229" watchObservedRunningTime="2026-10-30 05:40:31.922498856 +0000 UTC m=+486.030537890" Oct 30 05:40:32.011980 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.011949 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edefd0a-3b2a-48c7-b230-b5604a8a9d9c" path="/var/lib/kubelet/pods/9edefd0a-3b2a-48c7-b230-b5604a8a9d9c/volumes" Oct 30 05:40:32.044394 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.044333 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:32.079834 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.079806 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb\" (UniqueName: \"kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb\") pod \"0b421985-9d2b-474e-870d-20e1a6d416b1\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " Oct 30 05:40:32.079993 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.079911 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume\") pod \"0b421985-9d2b-474e-870d-20e1a6d416b1\" (UID: \"0b421985-9d2b-474e-870d-20e1a6d416b1\") " Oct 30 05:40:32.080332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.080309 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume" pod "0b421985-9d2b-474e-870d-20e1a6d416b1" (UID: "0b421985-9d2b-474e-870d-20e1a6d416b1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:32.082134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.082096 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb" pod "0b421985-9d2b-474e-870d-20e1a6d416b1" (UID: "0b421985-9d2b-474e-870d-20e1a6d416b1"). InnerVolumeSpecName "kube-api-access-5l2mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:32.180740 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.180706 5980 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b421985-9d2b-474e-870d-20e1a6d416b1-extensions-socket-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:32.180740 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.180734 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l2mb\" (UniqueName: \"kubernetes.io/projected/0b421985-9d2b-474e-870d-20e1a6d416b1-kube-api-access-5l2mb\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:32.789644 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.789604 5980 generic.go:356] "Generic (PLEG): container finished" podID="0b421985-9d2b-474e-870d-20e1a6d416b1" containerID="963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47" exitCode=0 Oct 30 05:40:32.790090 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.789662 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:32.790090 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.789713 5980 scope.go:122] "RemoveContainer" containerID="963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47" Oct 30 05:40:32.798618 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.798601 5980 scope.go:122] "RemoveContainer" containerID="963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47" Oct 30 05:40:32.798884 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:32.798865 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47\": container with ID starting with 963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47 not found: ID does not exist" containerID="963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47" Oct 30 05:40:32.798935 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:32.798893 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47"} err="failed to get container status \"963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47\": rpc error: code = NotFound desc = could not find container \"963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47\": container with ID starting with 963635eecb72da4461695573f3ddf4d06ca003d8038fbb356d9929226cbeed47 not found: ID does not exist" Oct 30 05:40:32.812798 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:32.812762 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-p7vsg\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-p7vsg" Oct 30 05:40:34.011420 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:34.011387 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b421985-9d2b-474e-870d-20e1a6d416b1" path="/var/lib/kubelet/pods/0b421985-9d2b-474e-870d-20e1a6d416b1/volumes" Oct 30 05:40:42.791892 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:42.791858 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:42.792306 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:42.791934 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:42.976277 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:42.976245 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm"] Oct 30 05:40:42.976514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:42.976481 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" podUID="e81b740b-aedd-428a-9b68-2b91238165e2" containerName="manager" containerID="cri-o://6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90" gracePeriod=10 Oct 30 05:40:43.086343 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:43.086307 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:40:43.217823 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.217799 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:43.274855 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.274827 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume\") pod \"e81b740b-aedd-428a-9b68-2b91238165e2\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " Oct 30 05:40:43.275016 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.274879 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb\" (UniqueName: \"kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb\") pod \"e81b740b-aedd-428a-9b68-2b91238165e2\" (UID: \"e81b740b-aedd-428a-9b68-2b91238165e2\") " Oct 30 05:40:43.275276 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.275253 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume" pod "e81b740b-aedd-428a-9b68-2b91238165e2" (UID: "e81b740b-aedd-428a-9b68-2b91238165e2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:43.277241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.277212 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb" pod "e81b740b-aedd-428a-9b68-2b91238165e2" (UID: "e81b740b-aedd-428a-9b68-2b91238165e2"). InnerVolumeSpecName "kube-api-access-vz8rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:43.328555 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.328524 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24"] Oct 30 05:40:43.328951 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.328932 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e81b740b-aedd-428a-9b68-2b91238165e2" containerName="manager" Oct 30 05:40:43.328951 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.328950 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81b740b-aedd-428a-9b68-2b91238165e2" containerName="manager" Oct 30 05:40:43.329141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.329022 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81b740b-aedd-428a-9b68-2b91238165e2" containerName="manager" Oct 30 05:40:43.332061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.332046 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.375527 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.375488 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnbh\" (UniqueName: \"kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m6j24\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.375727 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.375552 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m6j24\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.375727 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.375709 5980 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e81b740b-aedd-428a-9b68-2b91238165e2-extensions-socket-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:43.375821 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.375733 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz8rb\" (UniqueName: \"kubernetes.io/projected/e81b740b-aedd-428a-9b68-2b91238165e2-kube-api-access-vz8rb\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:43.376060 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.376044 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24"] Oct 30 05:40:43.476979 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.476945 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnbh\" (UniqueName: \"kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m6j24\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.477171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.476992 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m6j24\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.477379 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.477363 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m6j24\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.502819 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.502791 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnbh\" (UniqueName: \"kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh\") pod \"kuadrant-operator-controller-manager-57bb9bfff-m6j24\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.644419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.644332 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:43.838227 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.838193 5980 generic.go:356] "Generic (PLEG): container finished" podID="e81b740b-aedd-428a-9b68-2b91238165e2" containerID="6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90" exitCode=0 Oct 30 05:40:43.838656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.838245 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" Oct 30 05:40:43.838656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.838249 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" event={"ID":"e81b740b-aedd-428a-9b68-2b91238165e2","Type":"ContainerDied","Data":"6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90"} Oct 30 05:40:43.838656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.838292 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm" event={"ID":"e81b740b-aedd-428a-9b68-2b91238165e2","Type":"ContainerDied","Data":"7cfaec7a1c647ce641a1ce0cab9e03658fd0ecac0b66431850b5424625c49fe8"} Oct 30 05:40:43.838656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.838314 5980 scope.go:122] "RemoveContainer" containerID="6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90" Oct 30 05:40:43.848940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.848915 5980 scope.go:122] "RemoveContainer" containerID="6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90" Oct 30 05:40:43.849693 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:43.849671 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90\": container with ID starting with 6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90 not found: ID does not exist" containerID="6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90" Oct 30 05:40:43.849776 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.849704 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90"} err="failed to get container status \"6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90\": rpc error: code = NotFound desc = could not find container \"6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90\": container with ID starting with 6407bbf8aa9cc9c7479d0543984c902d89d826c11fa5c55281a6452b7005fb90 not found: ID does not exist" Oct 30 05:40:43.898337 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.898246 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm"] Oct 30 05:40:43.911777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:43.911736 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-9nbsm"] Oct 30 05:40:44.011340 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:44.011295 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81b740b-aedd-428a-9b68-2b91238165e2" path="/var/lib/kubelet/pods/e81b740b-aedd-428a-9b68-2b91238165e2/volumes" Oct 30 05:40:44.016387 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:44.016359 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24"] Oct 30 05:40:44.019334 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:40:44.019305 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b4ebd4_0abd_4b74_a016_900c456bf9e5.slice/crio-c7aa7cfc160301b88d903081e2fb3a20815ad96b8b54d84f860a3b79785b00b2 WatchSource:0}: Error finding container c7aa7cfc160301b88d903081e2fb3a20815ad96b8b54d84f860a3b79785b00b2: Status 404 returned error can't find the container with id c7aa7cfc160301b88d903081e2fb3a20815ad96b8b54d84f860a3b79785b00b2 Oct 30 05:40:44.845183 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:44.845148 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" event={"ID":"03b4ebd4-0abd-4b74-a016-900c456bf9e5","Type":"ContainerStarted","Data":"c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d"} Oct 30 05:40:44.845588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:44.845209 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" event={"ID":"03b4ebd4-0abd-4b74-a016-900c456bf9e5","Type":"ContainerStarted","Data":"c7aa7cfc160301b88d903081e2fb3a20815ad96b8b54d84f860a3b79785b00b2"} Oct 30 05:40:44.845588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:44.845288 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:44.901125 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:44.901065 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" podStartSLOduration=1.901051807 podStartE2EDuration="1.901051807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:44.898985945 +0000 UTC m=+499.007024977" watchObservedRunningTime="2026-10-30 05:40:44.901051807 +0000 UTC m=+499.009090839" Oct 30 05:40:55.850802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:55.850769 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 05:40:55.960258 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:55.960220 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6"] Oct 30 05:40:55.960492 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:55.960449 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" podUID="12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" containerName="manager" containerID="cri-o://d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7" gracePeriod=10 Oct 30 05:40:56.694457 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.694426 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:56.793235 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.793196 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk\" (UniqueName: \"kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk\") pod \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " Oct 30 05:40:56.793408 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.793243 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume\") pod \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\" (UID: \"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1\") " Oct 30 05:40:56.793678 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.793657 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume" pod "12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" (UID: "12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:40:56.795583 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.795560 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk" pod "12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" (UID: "12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1"). InnerVolumeSpecName "kube-api-access-sb9wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:56.893428 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893339 5980 generic.go:356] "Generic (PLEG): container finished" podID="12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" containerID="d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7" exitCode=0 Oct 30 05:40:56.893428 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893392 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" Oct 30 05:40:56.893921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893431 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" event={"ID":"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1","Type":"ContainerDied","Data":"d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7"} Oct 30 05:40:56.893921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893494 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6" event={"ID":"12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1","Type":"ContainerDied","Data":"87c7762c9ba27d5f233d98b594f32fad576f0130fd3ae583e8f07456fada4158"} Oct 30 05:40:56.893921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893523 5980 scope.go:122] "RemoveContainer" containerID="d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7" Oct 30 05:40:56.893921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893855 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb9wk\" (UniqueName: \"kubernetes.io/projected/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-kube-api-access-sb9wk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:56.893921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.893875 5980 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1-extensions-socket-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:40:56.902521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.902500 5980 scope.go:122] "RemoveContainer" containerID="d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7" Oct 30 05:40:56.902809 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:40:56.902789 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7\": container with ID starting with d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7 not found: ID does not exist" containerID="d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7" Oct 30 05:40:56.902862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.902818 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7"} err="failed to get container status \"d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7\": rpc error: code = NotFound desc = could not find container \"d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7\": container with ID starting with d596d146231f300b79b546cdf2c38cabac8b86459f04907c36cc1d4dffdbc9e7 not found: ID does not exist" Oct 30 05:40:56.948553 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.948523 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6"] Oct 30 05:40:56.953093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:56.953068 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-crtp6"] Oct 30 05:40:58.011871 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:40:58.011833 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" path="/var/lib/kubelet/pods/12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1/volumes" Oct 30 05:41:13.087460 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:41:13.087418 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:41:13.914193 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.914153 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lkfkb"] Oct 30 05:41:13.914586 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.914567 5980 cpu_manager.go:414] "containerMap: removing container" podUID="12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" containerName="manager" Oct 30 05:41:13.914681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.914588 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" containerName="manager" Oct 30 05:41:13.914681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.914677 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b33cf5-b1f6-44c0-93c6-f7ae6e6dc4c1" containerName="manager" Oct 30 05:41:13.917487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.917464 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:13.934576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.934549 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lkfkb"] Oct 30 05:41:13.947249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:13.947224 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sffvg\"" Oct 30 05:41:14.033025 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:14.032988 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvzk6\" (UniqueName: \"kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6\") pod \"authorino-57f7f8659f-lkfkb\" (UID: \"b0aff42d-1a69-46d9-9662-621f7971892a\") " pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:14.134101 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:14.134060 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvzk6\" (UniqueName: \"kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6\") pod \"authorino-57f7f8659f-lkfkb\" (UID: \"b0aff42d-1a69-46d9-9662-621f7971892a\") " pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:14.161738 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:14.161714 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvzk6\" (UniqueName: \"kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6\") pod \"authorino-57f7f8659f-lkfkb\" (UID: \"b0aff42d-1a69-46d9-9662-621f7971892a\") " pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:14.229677 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:14.229581 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:14.523342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:14.523274 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lkfkb"] Oct 30 05:41:14.526930 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:41:14.526905 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0aff42d_1a69_46d9_9662_621f7971892a.slice/crio-1245bda4f0b8d12fa76d12cfcd7b8d9fb782323b68c177c7728782a677c0541f WatchSource:0}: Error finding container 1245bda4f0b8d12fa76d12cfcd7b8d9fb782323b68c177c7728782a677c0541f: Status 404 returned error can't find the container with id 1245bda4f0b8d12fa76d12cfcd7b8d9fb782323b68c177c7728782a677c0541f Oct 30 05:41:14.971306 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:14.971265 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" event={"ID":"b0aff42d-1a69-46d9-9662-621f7971892a","Type":"ContainerStarted","Data":"1245bda4f0b8d12fa76d12cfcd7b8d9fb782323b68c177c7728782a677c0541f"} Oct 30 05:41:15.977914 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:15.977877 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" event={"ID":"b0aff42d-1a69-46d9-9662-621f7971892a","Type":"ContainerStarted","Data":"042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1"} Oct 30 05:41:16.018701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:16.018645 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" podStartSLOduration=1.7649604 podStartE2EDuration="3.018627927s" totalImagesPullingTime="1.253667527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:16.018603496 +0000 UTC m=+530.126642529" watchObservedRunningTime="2026-10-30 05:41:16.018627927 +0000 UTC m=+530.126666958" Oct 30 05:41:17.333850 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:17.333816 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lkfkb"] Oct 30 05:41:17.985880 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:17.985843 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" podUID="b0aff42d-1a69-46d9-9662-621f7971892a" containerName="authorino" containerID="cri-o://042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1" gracePeriod=30 Oct 30 05:41:18.231352 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.231326 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:18.267459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.267378 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6\" (UniqueName: \"kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6\") pod \"b0aff42d-1a69-46d9-9662-621f7971892a\" (UID: \"b0aff42d-1a69-46d9-9662-621f7971892a\") " Oct 30 05:41:18.269600 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.269571 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6" pod "b0aff42d-1a69-46d9-9662-621f7971892a" (UID: "b0aff42d-1a69-46d9-9662-621f7971892a"). InnerVolumeSpecName "kube-api-access-lvzk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:41:18.368022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.367987 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvzk6\" (UniqueName: \"kubernetes.io/projected/b0aff42d-1a69-46d9-9662-621f7971892a-kube-api-access-lvzk6\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:41:18.992005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.991973 5980 generic.go:356] "Generic (PLEG): container finished" podID="b0aff42d-1a69-46d9-9662-621f7971892a" containerID="042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1" exitCode=0 Oct 30 05:41:18.992210 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.992017 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" Oct 30 05:41:18.992210 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.992061 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" event={"ID":"b0aff42d-1a69-46d9-9662-621f7971892a","Type":"ContainerDied","Data":"042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1"} Oct 30 05:41:18.992210 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.992099 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lkfkb" event={"ID":"b0aff42d-1a69-46d9-9662-621f7971892a","Type":"ContainerDied","Data":"1245bda4f0b8d12fa76d12cfcd7b8d9fb782323b68c177c7728782a677c0541f"} Oct 30 05:41:18.992210 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:18.992127 5980 scope.go:122] "RemoveContainer" containerID="042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1" Oct 30 05:41:19.000187 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:19.000164 5980 scope.go:122] "RemoveContainer" containerID="042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1" Oct 30 05:41:19.000494 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:41:19.000476 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1\": container with ID starting with 042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1 not found: ID does not exist" containerID="042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1" Oct 30 05:41:19.000543 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:19.000505 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1"} err="failed to get container status \"042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1\": rpc error: code = NotFound desc = could not find container \"042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1\": container with ID starting with 042d039032848e95b405a2a0699f0974f52d462269189259fd7a38aca68f64f1 not found: ID does not exist" Oct 30 05:41:19.046182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:19.046150 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lkfkb"] Oct 30 05:41:19.052158 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:19.052131 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lkfkb"] Oct 30 05:41:20.010718 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:20.010686 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0aff42d-1a69-46d9-9662-621f7971892a" path="/var/lib/kubelet/pods/b0aff42d-1a69-46d9-9662-621f7971892a/volumes" Oct 30 05:41:33.136154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.136099 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4"] Oct 30 05:41:33.136613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.136425 5980 cpu_manager.go:414] "containerMap: removing container" podUID="b0aff42d-1a69-46d9-9662-621f7971892a" containerName="authorino" Oct 30 05:41:33.136613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.136435 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0aff42d-1a69-46d9-9662-621f7971892a" containerName="authorino" Oct 30 05:41:33.136613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.136485 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0aff42d-1a69-46d9-9662-621f7971892a" containerName="authorino" Oct 30 05:41:33.141063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.141042 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.158301 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.158275 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4"] Oct 30 05:41:33.160529 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.160511 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 05:41:33.171074 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.171046 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xgmfz\"" Oct 30 05:41:33.171074 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.171062 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:41:33.171300 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.171147 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:41:33.174453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.174423 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0f53d1a-6fce-4313-b6b9-3153e29f7f13-data\") pod \"postgres-7cdbf6dc44-p64x4\" (UID: \"b0f53d1a-6fce-4313-b6b9-3153e29f7f13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.174576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.174461 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl72r\" (UniqueName: \"kubernetes.io/projected/b0f53d1a-6fce-4313-b6b9-3153e29f7f13-kube-api-access-vl72r\") pod \"postgres-7cdbf6dc44-p64x4\" (UID: \"b0f53d1a-6fce-4313-b6b9-3153e29f7f13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.275752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.275728 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0f53d1a-6fce-4313-b6b9-3153e29f7f13-data\") pod \"postgres-7cdbf6dc44-p64x4\" (UID: \"b0f53d1a-6fce-4313-b6b9-3153e29f7f13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.275870 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.275758 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl72r\" (UniqueName: \"kubernetes.io/projected/b0f53d1a-6fce-4313-b6b9-3153e29f7f13-kube-api-access-vl72r\") pod \"postgres-7cdbf6dc44-p64x4\" (UID: \"b0f53d1a-6fce-4313-b6b9-3153e29f7f13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.276090 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.276068 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b0f53d1a-6fce-4313-b6b9-3153e29f7f13-data\") pod \"postgres-7cdbf6dc44-p64x4\" (UID: \"b0f53d1a-6fce-4313-b6b9-3153e29f7f13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.300098 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.300074 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl72r\" (UniqueName: \"kubernetes.io/projected/b0f53d1a-6fce-4313-b6b9-3153e29f7f13-kube-api-access-vl72r\") pod \"postgres-7cdbf6dc44-p64x4\" (UID: \"b0f53d1a-6fce-4313-b6b9-3153e29f7f13\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.453078 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.453014 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:33.756390 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:33.756316 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4"] Oct 30 05:41:33.759343 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:41:33.759316 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f53d1a_6fce_4313_b6b9_3153e29f7f13.slice/crio-ef00b4e42c19c79753aa6212ec3ff5e90b9fac2b3fd2ce83f12c6de7942880a7 WatchSource:0}: Error finding container ef00b4e42c19c79753aa6212ec3ff5e90b9fac2b3fd2ce83f12c6de7942880a7: Status 404 returned error can't find the container with id ef00b4e42c19c79753aa6212ec3ff5e90b9fac2b3fd2ce83f12c6de7942880a7 Oct 30 05:41:34.053965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:34.053892 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" event={"ID":"b0f53d1a-6fce-4313-b6b9-3153e29f7f13","Type":"ContainerStarted","Data":"ef00b4e42c19c79753aa6212ec3ff5e90b9fac2b3fd2ce83f12c6de7942880a7"} Oct 30 05:41:39.076302 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:39.076259 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" event={"ID":"b0f53d1a-6fce-4313-b6b9-3153e29f7f13","Type":"ContainerStarted","Data":"d16170401c9c1a63ecc6d39545681f536331497364314f63723649dd5aadd3e6"} Oct 30 05:41:39.076727 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:39.076418 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:39.121605 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:39.121560 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" podStartSLOduration=1.352409161 podStartE2EDuration="6.121548881s" totalImagesPullingTime="4.76913972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:39.121199693 +0000 UTC m=+553.229238727" watchObservedRunningTime="2026-10-30 05:41:39.121548881 +0000 UTC m=+553.229587911" Oct 30 05:41:43.085741 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:41:43.085708 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:41:45.110197 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:45.110169 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" Oct 30 05:41:45.911145 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:45.911094 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8rmdr"] Oct 30 05:41:45.914214 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:45.914197 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:45.932491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:45.932469 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sffvg\"" Oct 30 05:41:45.946705 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:45.946682 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8rmdr"] Oct 30 05:41:45.980694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:45.980668 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcn6\" (UniqueName: \"kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6\") pod \"authorino-6f6944bcdd-8rmdr\" (UID: \"51d6cbcc-b507-4963-9e83-1ea15394f6b5\") " pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:46.081603 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.081570 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcn6\" (UniqueName: \"kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6\") pod \"authorino-6f6944bcdd-8rmdr\" (UID: \"51d6cbcc-b507-4963-9e83-1ea15394f6b5\") " pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:46.083936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.083914 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8rmdr"] Oct 30 05:41:46.084089 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:41:46.084072 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xlcn6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" podUID="51d6cbcc-b507-4963-9e83-1ea15394f6b5" Oct 30 05:41:46.102037 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.102019 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:46.108882 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.108860 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:46.109834 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.109813 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcn6\" (UniqueName: \"kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6\") pod \"authorino-6f6944bcdd-8rmdr\" (UID: \"51d6cbcc-b507-4963-9e83-1ea15394f6b5\") " pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:46.182355 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.182297 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6\" (UniqueName: \"kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6\") pod \"51d6cbcc-b507-4963-9e83-1ea15394f6b5\" (UID: \"51d6cbcc-b507-4963-9e83-1ea15394f6b5\") " Oct 30 05:41:46.184384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.184360 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6" pod "51d6cbcc-b507-4963-9e83-1ea15394f6b5" (UID: "51d6cbcc-b507-4963-9e83-1ea15394f6b5"). InnerVolumeSpecName "kube-api-access-xlcn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:41:46.282966 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:46.282941 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcn6\" (UniqueName: \"kubernetes.io/projected/51d6cbcc-b507-4963-9e83-1ea15394f6b5-kube-api-access-xlcn6\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:41:47.106005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:47.105974 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-8rmdr" Oct 30 05:41:47.189580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:47.189546 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8rmdr"] Oct 30 05:41:47.197636 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:47.197611 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-8rmdr"] Oct 30 05:41:48.010964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:41:48.010933 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d6cbcc-b507-4963-9e83-1ea15394f6b5" path="/var/lib/kubelet/pods/51d6cbcc-b507-4963-9e83-1ea15394f6b5/volumes" Oct 30 05:42:13.085835 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:42:13.085785 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:42:33.718810 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.718771 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-skq9m"] Oct 30 05:42:33.722349 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.722333 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.746465 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.746440 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.746597 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.746497 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.746597 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.746563 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tscc\" (UniqueName: \"kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.746674 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.746593 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.762332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.757966 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 05:42:33.762332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.757972 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4vtg8\"" Oct 30 05:42:33.762332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.758067 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:42:33.783826 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.783797 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-skq9m"] Oct 30 05:42:33.846953 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.846919 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tscc\" (UniqueName: \"kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.847165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.846968 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.847165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.847018 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.847165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.847088 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.847770 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.847746 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.849631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.849611 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.849734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.849655 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:33.878548 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:33.878522 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tscc\" (UniqueName: \"kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc\") pod \"maas-api-5c48877c75-skq9m\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:34.034812 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:34.034732 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:34.336573 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:34.336535 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-skq9m"] Oct 30 05:42:34.339456 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:42:34.339431 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eda9750_e242_40c3_984c_437d32d41f47.slice/crio-c1207101f40727a48e591c9974811f10e33f019574fb77a6baf61d9c6574b726 WatchSource:0}: Error finding container c1207101f40727a48e591c9974811f10e33f019574fb77a6baf61d9c6574b726: Status 404 returned error can't find the container with id c1207101f40727a48e591c9974811f10e33f019574fb77a6baf61d9c6574b726 Oct 30 05:42:34.340762 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:34.340744 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:42:35.293660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:35.293617 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" event={"ID":"5eda9750-e242-40c3-984c-437d32d41f47","Type":"ContainerStarted","Data":"c1207101f40727a48e591c9974811f10e33f019574fb77a6baf61d9c6574b726"} Oct 30 05:42:37.303245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:37.303176 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" event={"ID":"5eda9750-e242-40c3-984c-437d32d41f47","Type":"ContainerStarted","Data":"a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e"} Oct 30 05:42:37.303597 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:37.303302 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:42:37.358797 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:37.358738 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" podStartSLOduration=1.729552495 podStartE2EDuration="4.358720693s" totalImagesPullingTime="2.629168198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:37.357331584 +0000 UTC m=+611.465370618" watchObservedRunningTime="2026-10-30 05:42:37.358720693 +0000 UTC m=+611.466759739" Oct 30 05:42:43.086213 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:42:43.086178 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:42:43.313121 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:42:43.313077 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 05:43:13.085923 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:43:13.085887 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:43:42.557467 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.557437 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj"] Oct 30 05:43:42.561031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.561008 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.577770 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.577701 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 05:43:42.577770 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.577731 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2k6w7\"" Oct 30 05:43:42.591459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.591431 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 05:43:42.591634 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.591617 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 05:43:42.592949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.592928 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj"] Oct 30 05:43:42.599831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.595543 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.599831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.595617 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.599831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.595651 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rct\" (UniqueName: \"kubernetes.io/projected/3c016552-431e-4686-af6b-983057f85c80-kube-api-access-l2rct\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.599831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.595693 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c016552-431e-4686-af6b-983057f85c80-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.599831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.595751 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.599831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.595790 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697284 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697513 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697336 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697513 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697356 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rct\" (UniqueName: \"kubernetes.io/projected/3c016552-431e-4686-af6b-983057f85c80-kube-api-access-l2rct\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697513 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697382 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c016552-431e-4686-af6b-983057f85c80-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697513 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697428 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697513 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697449 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697806 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697781 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697806 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697797 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.697977 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.697954 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.699873 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.699853 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c016552-431e-4686-af6b-983057f85c80-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.700083 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.700065 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c016552-431e-4686-af6b-983057f85c80-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.722505 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.722479 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rct\" (UniqueName: \"kubernetes.io/projected/3c016552-431e-4686-af6b-983057f85c80-kube-api-access-l2rct\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj\" (UID: \"3c016552-431e-4686-af6b-983057f85c80\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:42.873876 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:42.873835 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:43.085823 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:43:43.085781 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:43:43.175565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:43.175409 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj"] Oct 30 05:43:43.177778 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:43:43.177750 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c016552_431e_4686_af6b_983057f85c80.slice/crio-0bcca1eee6f15d9a891dc3268bdd4d396059320f93b8d9835d20c9b100f5c98d WatchSource:0}: Error finding container 0bcca1eee6f15d9a891dc3268bdd4d396059320f93b8d9835d20c9b100f5c98d: Status 404 returned error can't find the container with id 0bcca1eee6f15d9a891dc3268bdd4d396059320f93b8d9835d20c9b100f5c98d Oct 30 05:43:43.558705 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:43.558629 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" event={"ID":"3c016552-431e-4686-af6b-983057f85c80","Type":"ContainerStarted","Data":"0bcca1eee6f15d9a891dc3268bdd4d396059320f93b8d9835d20c9b100f5c98d"} Oct 30 05:43:44.564395 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:44.564318 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" event={"ID":"3c016552-431e-4686-af6b-983057f85c80","Type":"ContainerStarted","Data":"78ccf0d6f1906adb82e104537b54c9bd5f8fb2e2a23bffdecfdc8c5572e97fd4"} Oct 30 05:43:44.611906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:44.611856 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" podStartSLOduration=1.5586616599999998 podStartE2EDuration="2.611843386s" totalImagesPullingTime="1.053181726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:44.611681153 +0000 UTC m=+678.719720180" watchObservedRunningTime="2026-10-30 05:43:44.611843386 +0000 UTC m=+678.719882423" Oct 30 05:43:44.874672 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:44.874647 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:46.341247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.341216 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5"] Oct 30 05:43:46.344524 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.344506 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.362800 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.362767 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 05:43:46.376589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.376567 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5"] Oct 30 05:43:46.429761 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.429734 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.429904 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.429774 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7fl\" (UniqueName: \"kubernetes.io/projected/206d662f-b8a8-48b5-b8f9-c74d9785870f-kube-api-access-th7fl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.429981 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.429929 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.429981 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.429975 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.430085 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.430069 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206d662f-b8a8-48b5-b8f9-c74d9785870f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.430173 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.430151 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531496 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531460 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531524 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531555 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7fl\" (UniqueName: \"kubernetes.io/projected/206d662f-b8a8-48b5-b8f9-c74d9785870f-kube-api-access-th7fl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531747 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531640 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531747 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531689 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531747 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531724 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206d662f-b8a8-48b5-b8f9-c74d9785870f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531895 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531874 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.531953 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531931 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.532016 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.531998 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.533968 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.533947 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/206d662f-b8a8-48b5-b8f9-c74d9785870f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.534191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.534173 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/206d662f-b8a8-48b5-b8f9-c74d9785870f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.557118 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.557086 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7fl\" (UniqueName: \"kubernetes.io/projected/206d662f-b8a8-48b5-b8f9-c74d9785870f-kube-api-access-th7fl\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5\" (UID: \"206d662f-b8a8-48b5-b8f9-c74d9785870f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.657120 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.657029 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:46.958317 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:46.958239 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5"] Oct 30 05:43:46.961003 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:43:46.960972 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod206d662f_b8a8_48b5_b8f9_c74d9785870f.slice/crio-abeacc0429ccfcce32463f870f6b92e38df2f1df9ab643d2293261f3b701e152 WatchSource:0}: Error finding container abeacc0429ccfcce32463f870f6b92e38df2f1df9ab643d2293261f3b701e152: Status 404 returned error can't find the container with id abeacc0429ccfcce32463f870f6b92e38df2f1df9ab643d2293261f3b701e152 Oct 30 05:43:47.579495 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:47.579410 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" event={"ID":"206d662f-b8a8-48b5-b8f9-c74d9785870f","Type":"ContainerStarted","Data":"843064c1af93e7b40c6d9ce832a2574ea8ae25456d05ba8ce065b0a725fae433"} Oct 30 05:43:47.579495 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:47.579467 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" event={"ID":"206d662f-b8a8-48b5-b8f9-c74d9785870f","Type":"ContainerStarted","Data":"abeacc0429ccfcce32463f870f6b92e38df2f1df9ab643d2293261f3b701e152"} Oct 30 05:43:47.624717 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:47.624665 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podStartSLOduration=1.46148675 podStartE2EDuration="1.624650713s" totalImagesPullingTime="163.163963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:47.623859194 +0000 UTC m=+681.731898226" watchObservedRunningTime="2026-10-30 05:43:47.624650713 +0000 UTC m=+681.732689767" Oct 30 05:43:47.657254 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:47.657227 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:48.435582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.435555 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676"] Oct 30 05:43:48.438937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.438916 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.455975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.455952 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 05:43:48.469962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.469938 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676"] Oct 30 05:43:48.549041 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.549011 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.549206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.549057 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.549206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.549144 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.549206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.549169 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.549309 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.549246 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxhs\" (UniqueName: \"kubernetes.io/projected/2f109c85-5be4-41f3-b199-0d6da6a35a82-kube-api-access-bxxhs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.549309 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.549287 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f109c85-5be4-41f3-b199-0d6da6a35a82-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650034 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650068 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650084 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650150 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxhs\" (UniqueName: \"kubernetes.io/projected/2f109c85-5be4-41f3-b199-0d6da6a35a82-kube-api-access-bxxhs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650182 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f109c85-5be4-41f3-b199-0d6da6a35a82-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650346 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650497 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650645 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.650900 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.650879 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.652611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.652586 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f109c85-5be4-41f3-b199-0d6da6a35a82-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.652891 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.652871 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f109c85-5be4-41f3-b199-0d6da6a35a82-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.680239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.680213 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxhs\" (UniqueName: \"kubernetes.io/projected/2f109c85-5be4-41f3-b199-0d6da6a35a82-kube-api-access-bxxhs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zx676\" (UID: \"2f109c85-5be4-41f3-b199-0d6da6a35a82\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:48.751885 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:48.751821 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:49.054209 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:49.054120 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676"] Oct 30 05:43:49.056793 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:43:49.056768 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f109c85_5be4_41f3_b199_0d6da6a35a82.slice/crio-2ab398e94111b0e4ca3c39295aacfc28fb4d5e23f1e941fdfc65cfd1bcbdf0bd WatchSource:0}: Error finding container 2ab398e94111b0e4ca3c39295aacfc28fb4d5e23f1e941fdfc65cfd1bcbdf0bd: Status 404 returned error can't find the container with id 2ab398e94111b0e4ca3c39295aacfc28fb4d5e23f1e941fdfc65cfd1bcbdf0bd Oct 30 05:43:49.589211 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:49.589174 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" event={"ID":"2f109c85-5be4-41f3-b199-0d6da6a35a82","Type":"ContainerStarted","Data":"dab214997ee3a26e72b7a2c7d00037bee99d0b99eaef5170a30307eca06767be"} Oct 30 05:43:49.589407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:49.589227 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" event={"ID":"2f109c85-5be4-41f3-b199-0d6da6a35a82","Type":"ContainerStarted","Data":"2ab398e94111b0e4ca3c39295aacfc28fb4d5e23f1e941fdfc65cfd1bcbdf0bd"} Oct 30 05:43:49.634865 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:49.634814 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" podStartSLOduration=1.476718805 podStartE2EDuration="1.634800251s" totalImagesPullingTime="158.081446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:49.634498457 +0000 UTC m=+683.742537490" watchObservedRunningTime="2026-10-30 05:43:49.634800251 +0000 UTC m=+683.742839283" Oct 30 05:43:49.752156 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:49.752128 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:52.875039 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:52.875006 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:52.887543 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:52.887513 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:53.615209 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:53.615182 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj" Oct 30 05:43:56.657600 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:56.657570 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:56.670376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:56.670353 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:57.636867 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:57.636830 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:43:58.752646 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:58.752612 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:58.764955 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:58.764935 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:43:59.645200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:43:59.645177 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zx676" Oct 30 05:44:06.496503 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.496469 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54784db4fd-vzlfm"] Oct 30 05:44:06.500058 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.500036 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.518805 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.518780 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54784db4fd-vzlfm"] Oct 30 05:44:06.531125 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.531082 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 05:44:06.531250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.531199 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sffvg\"" Oct 30 05:44:06.602320 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.602292 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2mr\" (UniqueName: \"kubernetes.io/projected/dd876cb0-6d24-4d4f-9a21-65cf65d92188-kube-api-access-rt2mr\") pod \"authorino-54784db4fd-vzlfm\" (UID: \"dd876cb0-6d24-4d4f-9a21-65cf65d92188\") " pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.602494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.602331 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd876cb0-6d24-4d4f-9a21-65cf65d92188-tls-cert\") pod \"authorino-54784db4fd-vzlfm\" (UID: \"dd876cb0-6d24-4d4f-9a21-65cf65d92188\") " pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.703002 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.702964 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2mr\" (UniqueName: \"kubernetes.io/projected/dd876cb0-6d24-4d4f-9a21-65cf65d92188-kube-api-access-rt2mr\") pod \"authorino-54784db4fd-vzlfm\" (UID: \"dd876cb0-6d24-4d4f-9a21-65cf65d92188\") " pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.703225 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.703018 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd876cb0-6d24-4d4f-9a21-65cf65d92188-tls-cert\") pod \"authorino-54784db4fd-vzlfm\" (UID: \"dd876cb0-6d24-4d4f-9a21-65cf65d92188\") " pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.705922 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.705885 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd876cb0-6d24-4d4f-9a21-65cf65d92188-tls-cert\") pod \"authorino-54784db4fd-vzlfm\" (UID: \"dd876cb0-6d24-4d4f-9a21-65cf65d92188\") " pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.728013 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.727984 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2mr\" (UniqueName: \"kubernetes.io/projected/dd876cb0-6d24-4d4f-9a21-65cf65d92188-kube-api-access-rt2mr\") pod \"authorino-54784db4fd-vzlfm\" (UID: \"dd876cb0-6d24-4d4f-9a21-65cf65d92188\") " pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:06.811874 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:06.811814 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54784db4fd-vzlfm" Oct 30 05:44:07.107918 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:07.107880 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54784db4fd-vzlfm"] Oct 30 05:44:07.110526 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:44:07.110483 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd876cb0_6d24_4d4f_9a21_65cf65d92188.slice/crio-5ac4126e28f353e5b7af392e143759806f1594edabd07e4f76a6c25ed3ed4e17 WatchSource:0}: Error finding container 5ac4126e28f353e5b7af392e143759806f1594edabd07e4f76a6c25ed3ed4e17: Status 404 returned error can't find the container with id 5ac4126e28f353e5b7af392e143759806f1594edabd07e4f76a6c25ed3ed4e17 Oct 30 05:44:07.670606 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:07.670575 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54784db4fd-vzlfm" event={"ID":"dd876cb0-6d24-4d4f-9a21-65cf65d92188","Type":"ContainerStarted","Data":"14e27787e46d67e6d9afecdda72478e9416659ac324e5e61c6c5fee27a10fa2b"} Oct 30 05:44:07.670958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:07.670615 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54784db4fd-vzlfm" event={"ID":"dd876cb0-6d24-4d4f-9a21-65cf65d92188","Type":"ContainerStarted","Data":"5ac4126e28f353e5b7af392e143759806f1594edabd07e4f76a6c25ed3ed4e17"} Oct 30 05:44:07.721754 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:07.721705 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-54784db4fd-vzlfm" podStartSLOduration=1.32695321 podStartE2EDuration="1.721692459s" totalImagesPullingTime="394.739249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:07.720761184 +0000 UTC m=+701.828800235" watchObservedRunningTime="2026-10-30 05:44:07.721692459 +0000 UTC m=+701.829731491" Oct 30 05:44:13.086577 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:44:13.086542 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:44:13.504583 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.504499 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz"] Oct 30 05:44:13.507728 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.507705 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.527617 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.527595 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 05:44:13.542272 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.542250 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz"] Oct 30 05:44:13.565964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.565927 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzpt\" (UniqueName: \"kubernetes.io/projected/5a0e5f24-1add-4aaf-ae9d-7bab12f8487b-kube-api-access-pvzpt\") pod \"payload-pre-processing-7b98bb678c-dlwvz\" (UID: \"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.566212 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.566153 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0e5f24-1add-4aaf-ae9d-7bab12f8487b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dlwvz\" (UID: \"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.666818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.666786 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0e5f24-1add-4aaf-ae9d-7bab12f8487b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dlwvz\" (UID: \"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.666954 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.666828 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzpt\" (UniqueName: \"kubernetes.io/projected/5a0e5f24-1add-4aaf-ae9d-7bab12f8487b-kube-api-access-pvzpt\") pod \"payload-pre-processing-7b98bb678c-dlwvz\" (UID: \"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.667535 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.667517 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0e5f24-1add-4aaf-ae9d-7bab12f8487b-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dlwvz\" (UID: \"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.696271 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.696244 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzpt\" (UniqueName: \"kubernetes.io/projected/5a0e5f24-1add-4aaf-ae9d-7bab12f8487b-kube-api-access-pvzpt\") pod \"payload-pre-processing-7b98bb678c-dlwvz\" (UID: \"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:13.823305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:13.823276 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:14.123801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:14.123644 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz"] Oct 30 05:44:14.126744 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:44:14.126717 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0e5f24_1add_4aaf_ae9d_7bab12f8487b.slice/crio-3d6bc00ff3636d6fd24ae11c397b05db9010a15c9f061e63ee0cac1be04b3640 WatchSource:0}: Error finding container 3d6bc00ff3636d6fd24ae11c397b05db9010a15c9f061e63ee0cac1be04b3640: Status 404 returned error can't find the container with id 3d6bc00ff3636d6fd24ae11c397b05db9010a15c9f061e63ee0cac1be04b3640 Oct 30 05:44:14.706509 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:14.706474 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" event={"ID":"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b","Type":"ContainerStarted","Data":"3d6bc00ff3636d6fd24ae11c397b05db9010a15c9f061e63ee0cac1be04b3640"} Oct 30 05:44:16.395749 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.395720 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh"] Oct 30 05:44:16.399294 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.399274 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.415285 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.415257 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 05:44:16.415398 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.415347 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ntk7q\"" Oct 30 05:44:16.430370 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.430346 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh"] Oct 30 05:44:16.491325 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491297 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491448 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491334 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491448 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491377 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk2nf\" (UniqueName: \"kubernetes.io/projected/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-kube-api-access-sk2nf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491448 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491415 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491448 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491435 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491596 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491502 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491637 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491619 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491642 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.491712 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.491702 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592197 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592172 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592204 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592232 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk2nf\" (UniqueName: \"kubernetes.io/projected/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-kube-api-access-sk2nf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592249 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592265 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592308 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592345 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592364 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592413 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592750 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592611 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592809 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592774 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592829 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.592863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.592856 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.593268 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.593247 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.594625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.594605 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.594873 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.594853 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.618682 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.618651 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.618836 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.618820 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk2nf\" (UniqueName: \"kubernetes.io/projected/ece567c8-3dcb-4fd2-9ff3-48da693b4b1e-kube-api-access-sk2nf\") pod \"maas-default-gateway-openshift-default-58847f9fc5-8r2sh\" (UID: \"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.711347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.711290 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:16.716012 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.715990 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" event={"ID":"5a0e5f24-1add-4aaf-ae9d-7bab12f8487b","Type":"ContainerStarted","Data":"0d46b1f7f954088eb2303de2e271c6efcb39895ea6b69e8bac2bbc5b1b89f63f"} Oct 30 05:44:16.716157 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.716137 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:16.760342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:16.760284 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" podStartSLOduration=1.821457819 podStartE2EDuration="3.760263478s" totalImagesPullingTime="1.938805659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:16.758980798 +0000 UTC m=+710.867019831" watchObservedRunningTime="2026-10-30 05:44:16.760263478 +0000 UTC m=+710.868302512" Oct 30 05:44:17.030350 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:17.030330 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh"] Oct 30 05:44:17.032325 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:44:17.032305 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece567c8_3dcb_4fd2_9ff3_48da693b4b1e.slice/crio-f1abb9f1724a7734c0d304685ab664202bf8090e45cbad5021fbdb5a1f022d7d WatchSource:0}: Error finding container f1abb9f1724a7734c0d304685ab664202bf8090e45cbad5021fbdb5a1f022d7d: Status 404 returned error can't find the container with id f1abb9f1724a7734c0d304685ab664202bf8090e45cbad5021fbdb5a1f022d7d Oct 30 05:44:17.722083 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:17.722051 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" event={"ID":"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e","Type":"ContainerStarted","Data":"f1abb9f1724a7734c0d304685ab664202bf8090e45cbad5021fbdb5a1f022d7d"} Oct 30 05:44:19.799252 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:19.799222 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:19.799509 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:19.799285 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:19.799509 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:19.799315 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:20.739225 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:20.739181 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" event={"ID":"ece567c8-3dcb-4fd2-9ff3-48da693b4b1e","Type":"ContainerStarted","Data":"45807bc13ea5e4f6832fa8aa127f96c1152f751f618d0aa6f16eb79303812413"} Oct 30 05:44:20.825591 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:20.825536 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podStartSLOduration=2.061331902 podStartE2EDuration="4.825518988s" totalImagesPullingTime="2.764187086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:20.81126119 +0000 UTC m=+714.919300254" watchObservedRunningTime="2026-10-30 05:44:20.825518988 +0000 UTC m=+714.933558021" Oct 30 05:44:21.712519 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:21.712485 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:21.713879 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:21.713857 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:21.713939 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:21.713912 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:22.712209 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:22.712178 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:22.712584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:22.712231 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:23.711656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:23.711618 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:23.711843 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:23.711670 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:24.711960 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:24.711912 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:24.712399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:24.711984 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:25.712373 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:25.712337 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:25.712768 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:25.712397 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:26.712298 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:26.712264 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:26.712455 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:26.712311 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:27.725572 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:27.725545 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dlwvz" Oct 30 05:44:27.967021 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:27.966994 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:27.967490 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:27.967426 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:27.968437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:27.968419 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" Oct 30 05:44:31.471396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.471361 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z"] Oct 30 05:44:31.474575 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.474557 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.505662 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.505631 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z"] Oct 30 05:44:31.648247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648221 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648250 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648277 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648376 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648417 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648481 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648514 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648594 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648541 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.648594 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.648564 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.749706 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.749647 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.749706 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.749684 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.749706 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.749701 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.749904 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.749729 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.749904 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.749762 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.749904 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.749800 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750018 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750069 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750073 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750131 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750262 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750391 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750475 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750454 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.750550 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.750533 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.752275 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.752257 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.752409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.752393 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.775735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.775713 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.778633 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.778608 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-7dz5z\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:31.787598 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:31.787580 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:32.102538 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.102509 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z"] Oct 30 05:44:32.105372 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:44:32.105346 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b5d1f1_2dfd_42c8_ab76_e5305ae134df.slice/crio-6911e41a5e7e7549f9c55ddd7afbc09080dc41fbfe724ee87d5fc16a6fdb7ae4 WatchSource:0}: Error finding container 6911e41a5e7e7549f9c55ddd7afbc09080dc41fbfe724ee87d5fc16a6fdb7ae4: Status 404 returned error can't find the container with id 6911e41a5e7e7549f9c55ddd7afbc09080dc41fbfe724ee87d5fc16a6fdb7ae4 Oct 30 05:44:32.107538 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.107507 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:32.107636 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.107584 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:32.107700 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.107640 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:32.791769 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.791734 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" event={"ID":"74b5d1f1-2dfd-42c8-ab76-e5305ae134df","Type":"ContainerStarted","Data":"7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f"} Oct 30 05:44:32.792194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.791807 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" event={"ID":"74b5d1f1-2dfd-42c8-ab76-e5305ae134df","Type":"ContainerStarted","Data":"6911e41a5e7e7549f9c55ddd7afbc09080dc41fbfe724ee87d5fc16a6fdb7ae4"} Oct 30 05:44:32.841260 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:32.841212 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podStartSLOduration=1.841199661 podStartE2EDuration="1.841199661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:32.840064698 +0000 UTC m=+726.948103730" watchObservedRunningTime="2026-10-30 05:44:32.841199661 +0000 UTC m=+726.949238760" Oct 30 05:44:33.788564 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:33.788530 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:33.789975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:33.789943 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:33.790124 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:33.789995 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:34.788516 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:34.788478 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:34.789017 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:34.788543 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:35.788737 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:35.788652 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:35.788737 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:35.788715 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:36.788770 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:36.788733 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:36.789195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:36.788794 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:37.788580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:37.788537 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:37.788753 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:37.788594 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:38.789016 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:38.788979 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:38.789510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:38.789041 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:40.294442 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:40.294407 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:40.294940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:40.294823 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:40.295690 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:44:40.295670 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 05:44:43.086415 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:44:43.086373 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:45:04.647821 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.647788 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh"] Oct 30 05:45:04.656908 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.656886 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.676633 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.676606 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh"] Oct 30 05:45:04.688296 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.688265 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-197e9b-openshift-default-dockercfg-w5q2q\"" Oct 30 05:45:04.751879 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.751852 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752029 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.751888 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2qr\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752029 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.751912 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752029 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.752018 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752211 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.752037 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752211 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.752061 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752211 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.752179 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.752284 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.752316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.752307 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853328 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853296 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853328 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853330 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853364 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853391 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2qr\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853424 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853458 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853481 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853513 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853566 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853924 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853904 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.853989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.853971 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.854046 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.854000 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.854178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.854158 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.854248 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.854206 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.855862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.855841 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.855945 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.855911 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.889289 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.889189 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2qr\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.890561 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.889917 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf"] Oct 30 05:45:04.893196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.893121 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:04.894640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.893676 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:04.919759 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.919738 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c78066-openshift-default-dockercfg-g958x\"" Oct 30 05:45:04.925952 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.925925 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf"] Oct 30 05:45:04.970755 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:04.970733 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:05.056207 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056174 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056345 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056281 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbkl\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056349 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056423 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056377 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056460 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056434 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056493 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056460 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056496 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056519 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.056629 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.056614 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158003 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.157956 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158003 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158008 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158263 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158195 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158263 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158230 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158440 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158418 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158531 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158458 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158531 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158484 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158629 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158536 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbkl\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158629 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158582 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158629 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158607 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158766 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158824 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.158923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.158902 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.159097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.159079 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.160724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.160704 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.161437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.161420 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.189800 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.189747 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.190782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.190751 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbkl\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.220900 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.220875 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:05.312695 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.312673 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh"] Oct 30 05:45:05.315173 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:05.315146 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936d2db7_98ed_4b19_bf3c_d302b57cf15e.slice/crio-cbf3d5859ddf5f1780143d77c2577ce62d79029cf28f07fe4066fdf2c749e305 WatchSource:0}: Error finding container cbf3d5859ddf5f1780143d77c2577ce62d79029cf28f07fe4066fdf2c749e305: Status 404 returned error can't find the container with id cbf3d5859ddf5f1780143d77c2577ce62d79029cf28f07fe4066fdf2c749e305 Oct 30 05:45:05.317435 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.317398 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.317537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.317477 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.317537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.317526 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.373426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.373402 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp"] Oct 30 05:45:05.376210 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.376193 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.393773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.393744 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1354a5-openshift-default-dockercfg-8c2kz\"" Oct 30 05:45:05.399869 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.399843 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp"] Oct 30 05:45:05.461415 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461335 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461415 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461372 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461415 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461409 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461586 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461630 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnvq\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461780 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461737 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461830 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461794 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461835 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.461954 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.461934 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.511823 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.511787 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm"] Oct 30 05:45:05.515579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.515088 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.528932 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.528907 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf"] Oct 30 05:45:05.533409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.533308 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-128436-openshift-default-dockercfg-dwwj9\"" Oct 30 05:45:05.535508 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:05.533466 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66025822_a5de_4e85_8d69_b7fb524b5858.slice/crio-d9a99b4b7d22ddb615849e8530993c9e42851d4863f44cf1c059181d974febb8 WatchSource:0}: Error finding container d9a99b4b7d22ddb615849e8530993c9e42851d4863f44cf1c059181d974febb8: Status 404 returned error can't find the container with id d9a99b4b7d22ddb615849e8530993c9e42851d4863f44cf1c059181d974febb8 Oct 30 05:45:05.539864 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.539827 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.539966 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.539908 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.539966 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.539949 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.542177 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.542067 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm"] Oct 30 05:45:05.563062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563040 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563074 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563099 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563176 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563201 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563243 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563302 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563334 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnvq\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563366 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563722 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563702 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.563972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.563952 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.564298 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.564274 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.564554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.564534 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.564626 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.564600 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.566750 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.566721 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.567048 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.567027 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.589334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.589302 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.589700 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.589676 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnvq\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.663954 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664001 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664144 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664184 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664290 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664391 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66khp\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664426 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664490 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.665426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.664566 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.694501 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.694472 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:05.766711 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766675 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.766851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766773 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.766851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766804 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.766921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766851 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.766921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766876 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.766921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766912 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.767010 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.766988 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66khp\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.767043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.767025 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.767074 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.767051 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.767825 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.767722 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.767825 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.767736 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.767960 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.767881 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.768422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.768403 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.768579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.768510 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.775074 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.773600 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.775346 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.775329 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.792274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.792250 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.792365 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.792250 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66khp\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.831201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.831177 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:05.931595 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.931561 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" event={"ID":"66025822-a5de-4e85-8d69-b7fb524b5858","Type":"ContainerStarted","Data":"5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217"} Oct 30 05:45:05.931790 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.931627 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" event={"ID":"66025822-a5de-4e85-8d69-b7fb524b5858","Type":"ContainerStarted","Data":"d9a99b4b7d22ddb615849e8530993c9e42851d4863f44cf1c059181d974febb8"} Oct 30 05:45:05.934118 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.934078 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" event={"ID":"936d2db7-98ed-4b19-bf3c-d302b57cf15e","Type":"ContainerStarted","Data":"fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe"} Oct 30 05:45:05.934223 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.934138 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" event={"ID":"936d2db7-98ed-4b19-bf3c-d302b57cf15e","Type":"ContainerStarted","Data":"cbf3d5859ddf5f1780143d77c2577ce62d79029cf28f07fe4066fdf2c749e305"} Oct 30 05:45:05.971669 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.971645 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:05.979636 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:05.979563 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podStartSLOduration=1.97955273 podStartE2EDuration="1.97955273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:05.978852211 +0000 UTC m=+760.086891244" watchObservedRunningTime="2026-10-30 05:45:05.97955273 +0000 UTC m=+760.087591763" Oct 30 05:45:06.004799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.004432 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp"] Oct 30 05:45:06.007566 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:06.007541 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8723f03_8296_4c76_b138_6375e3561d45.slice/crio-dc2329f34b4e63bc3a0280c64cd96c2acc68d3b17235798db1e94f9e23c681bf WatchSource:0}: Error finding container dc2329f34b4e63bc3a0280c64cd96c2acc68d3b17235798db1e94f9e23c681bf: Status 404 returned error can't find the container with id dc2329f34b4e63bc3a0280c64cd96c2acc68d3b17235798db1e94f9e23c681bf Oct 30 05:45:06.010053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.010014 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.010178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.010124 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.010243 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.010199 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.026877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.026829 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podStartSLOduration=2.026811329 podStartE2EDuration="2.026811329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:06.025626017 +0000 UTC m=+760.133665050" watchObservedRunningTime="2026-10-30 05:45:06.026811329 +0000 UTC m=+760.134850365" Oct 30 05:45:06.141608 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.141568 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm"] Oct 30 05:45:06.152183 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:06.151358 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2309786_9bb4_4274_962f_47f3697e5309.slice/crio-7e2b5c8287fc0e27be0a4bfb03e47e9494fc91664a2a894bf2b1f5c03fa020d0 WatchSource:0}: Error finding container 7e2b5c8287fc0e27be0a4bfb03e47e9494fc91664a2a894bf2b1f5c03fa020d0: Status 404 returned error can't find the container with id 7e2b5c8287fc0e27be0a4bfb03e47e9494fc91664a2a894bf2b1f5c03fa020d0 Oct 30 05:45:06.155006 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.154582 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.155006 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.154651 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.155006 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.154685 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.221760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.221737 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:06.228537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.228516 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:06.940628 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.940587 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" event={"ID":"f8723f03-8296-4c76-b138-6375e3561d45","Type":"ContainerStarted","Data":"9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc"} Oct 30 05:45:06.941073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.940680 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" event={"ID":"f8723f03-8296-4c76-b138-6375e3561d45","Type":"ContainerStarted","Data":"dc2329f34b4e63bc3a0280c64cd96c2acc68d3b17235798db1e94f9e23c681bf"} Oct 30 05:45:06.942138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.942089 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" event={"ID":"e2309786-9bb4-4274-962f-47f3697e5309","Type":"ContainerStarted","Data":"24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94"} Oct 30 05:45:06.942517 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.942495 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:06.942630 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.942544 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" event={"ID":"e2309786-9bb4-4274-962f-47f3697e5309","Type":"ContainerStarted","Data":"7e2b5c8287fc0e27be0a4bfb03e47e9494fc91664a2a894bf2b1f5c03fa020d0"} Oct 30 05:45:06.943148 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.943132 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:06.976650 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:06.976620 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:07.001858 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.001802 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podStartSLOduration=2.001783089 podStartE2EDuration="2.001783089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:06.987215622 +0000 UTC m=+761.095254674" watchObservedRunningTime="2026-10-30 05:45:07.001783089 +0000 UTC m=+761.109822125" Oct 30 05:45:07.088603 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.088546 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podStartSLOduration=2.088528527 podStartE2EDuration="2.088528527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:07.087101956 +0000 UTC m=+761.195140990" watchObservedRunningTime="2026-10-30 05:45:07.088528527 +0000 UTC m=+761.196567560" Oct 30 05:45:07.694868 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.694831 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:07.700771 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.700744 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:07.832198 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.832166 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:07.837074 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.837053 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:07.948199 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.948093 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:07.948199 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.948178 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:07.948708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.948204 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:07.948708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.948605 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:07.948818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.948795 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:07.949196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:07.949180 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:10.115755 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.115712 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp"] Oct 30 05:45:10.119434 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.119410 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.142820 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.142791 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp"] Oct 30 05:45:10.150735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.150708 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-c78066-serving-cert\"" Oct 30 05:45:10.213137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.213083 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.213299 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.213201 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.213299 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.213230 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4pr\" (UniqueName: \"kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.213299 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.213255 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.314694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.314663 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.314694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.314704 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.314965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.314719 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4pr\" (UniqueName: \"kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.314965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.314761 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.315515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.315493 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.317498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.317471 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.317600 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.317531 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.340356 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.340319 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4pr\" (UniqueName: \"kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr\") pod \"maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.434482 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.434398 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:10.746221 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.746197 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp"] Oct 30 05:45:10.748139 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:10.748073 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f875821_6c66_418b_b792_a749ad1e158a.slice/crio-4c5f363195e7b6a75b18bdae5f855b023628e579cad5e1a460356b99f5e68ac1 WatchSource:0}: Error finding container 4c5f363195e7b6a75b18bdae5f855b023628e579cad5e1a460356b99f5e68ac1: Status 404 returned error can't find the container with id 4c5f363195e7b6a75b18bdae5f855b023628e579cad5e1a460356b99f5e68ac1 Oct 30 05:45:10.963876 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:10.963844 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" event={"ID":"7f875821-6c66-418b-b792-a749ad1e158a","Type":"ContainerStarted","Data":"4c5f363195e7b6a75b18bdae5f855b023628e579cad5e1a460356b99f5e68ac1"} Oct 30 05:45:11.972919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:11.972878 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" event={"ID":"7f875821-6c66-418b-b792-a749ad1e158a","Type":"ContainerStarted","Data":"82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b"} Oct 30 05:45:11.976048 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:11.973797 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:12.021893 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:12.021851 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" podStartSLOduration=1.6445294750000001 podStartE2EDuration="2.021835674s" totalImagesPullingTime="377.306199ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:12.020849501 +0000 UTC m=+766.128888736" watchObservedRunningTime="2026-10-30 05:45:12.021835674 +0000 UTC m=+766.129874738" Oct 30 05:45:13.087243 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:45:13.087196 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:45:13.751014 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.750976 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd"] Oct 30 05:45:13.754604 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.754585 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.787052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.787019 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-197e9b-serving-cert\"" Oct 30 05:45:13.789399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.789372 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd"] Oct 30 05:45:13.860461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.860427 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.860461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.860460 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.860675 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.860529 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.860675 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.860574 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kt4\" (UniqueName: \"kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.963188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.962102 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.963188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.962199 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.963188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.962252 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.963188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.962295 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kt4\" (UniqueName: \"kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.973131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.972336 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.975893 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.975865 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:13.998776 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:13.998690 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:14.004760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:14.004691 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kt4\" (UniqueName: \"kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4\") pod \"maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:14.068765 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:14.068734 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:14.374637 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:14.374438 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd"] Oct 30 05:45:15.005404 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.005326 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" event={"ID":"3d31a88e-bac3-4483-85bd-146ab72eb228","Type":"ContainerStarted","Data":"2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d"} Oct 30 05:45:15.005404 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.005372 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" event={"ID":"3d31a88e-bac3-4483-85bd-146ab72eb228","Type":"ContainerStarted","Data":"013eb6b61253c36ff0878fd8863839f6e1de491710cd6702517146d3eebb81cd"} Oct 30 05:45:15.005581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.005461 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:15.097145 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.097065 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" podStartSLOduration=1.740799454 podStartE2EDuration="2.097051742s" totalImagesPullingTime="356.252288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:15.096377592 +0000 UTC m=+769.204416625" watchObservedRunningTime="2026-10-30 05:45:15.097051742 +0000 UTC m=+769.205090806" Oct 30 05:45:15.510854 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.510815 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr"] Oct 30 05:45:15.515208 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.515187 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.532429 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.532404 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4a68a1\"" Oct 30 05:45:15.532553 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.532449 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4a68a1-dockercfg-pv6td\"" Oct 30 05:45:15.534019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.533995 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr"] Oct 30 05:45:15.693726 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.693685 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.693910 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.693807 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4wl\" (UniqueName: \"kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl\") pod \"payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.795094 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.794511 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.795094 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.794578 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4wl\" (UniqueName: \"kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl\") pod \"payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.797353 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.795361 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.821910 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.821885 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4wl\" (UniqueName: \"kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl\") pod \"payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:15.829779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:15.829758 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:16.150475 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.150455 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr"] Oct 30 05:45:16.152470 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:16.152439 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d18805d_84cd_4b64_b910_0c0b14325230.slice/crio-f53d3f755d61b68d36ebc13aa7739ceaf8624f313fa85724bf0112a72ae227bb WatchSource:0}: Error finding container f53d3f755d61b68d36ebc13aa7739ceaf8624f313fa85724bf0112a72ae227bb: Status 404 returned error can't find the container with id f53d3f755d61b68d36ebc13aa7739ceaf8624f313fa85724bf0112a72ae227bb Oct 30 05:45:16.834334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.834295 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6"] Oct 30 05:45:16.837922 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.837900 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:16.858625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.858584 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-1354a5-dockercfg-x9dmh\"" Oct 30 05:45:16.860174 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.860150 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6"] Oct 30 05:45:16.903656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.903626 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:16.903851 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.903675 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75fj\" (UniqueName: \"kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj\") pod \"payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:16.946872 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:16.946848 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-1354a5\"" Oct 30 05:45:17.004901 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.004868 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:17.005250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.005226 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75fj\" (UniqueName: \"kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj\") pod \"payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:17.006672 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.006649 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:17.018801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.018766 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" event={"ID":"4d18805d-84cd-4b64-b910-0c0b14325230","Type":"ContainerStarted","Data":"d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837"} Oct 30 05:45:17.019001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.018828 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" event={"ID":"4d18805d-84cd-4b64-b910-0c0b14325230","Type":"ContainerStarted","Data":"f53d3f755d61b68d36ebc13aa7739ceaf8624f313fa85724bf0112a72ae227bb"} Oct 30 05:45:17.019419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.019253 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:17.032464 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.032440 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75fj\" (UniqueName: \"kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj\") pod \"payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:17.065139 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.064234 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" podStartSLOduration=2.064215607 podStartE2EDuration="2.064215607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:17.061834662 +0000 UTC m=+771.169873724" watchObservedRunningTime="2026-10-30 05:45:17.064215607 +0000 UTC m=+771.172254662" Oct 30 05:45:17.151494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.151413 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:17.505035 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.505011 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6"] Oct 30 05:45:17.508261 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:17.508235 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda4eb6a_85f1_428e_8193_33da6f1482e4.slice/crio-a71a5c0ba3a64774570443671014654babf5bc985430a96838ad163ce57010c2 WatchSource:0}: Error finding container a71a5c0ba3a64774570443671014654babf5bc985430a96838ad163ce57010c2: Status 404 returned error can't find the container with id a71a5c0ba3a64774570443671014654babf5bc985430a96838ad163ce57010c2 Oct 30 05:45:17.552038 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.552012 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk"] Oct 30 05:45:17.557484 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.557465 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.591140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.585842 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk"] Oct 30 05:45:17.673130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.667306 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-e2e3d0-serving-cert\"" Oct 30 05:45:17.712264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.712231 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.712437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.712296 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.712437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.712418 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9zc\" (UniqueName: \"kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.712581 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.712567 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.813554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.813472 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.813554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.813517 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9zc\" (UniqueName: \"kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.813773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.813583 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.813773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.813624 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.814345 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.814321 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.816020 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.815997 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.816222 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.816204 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.837546 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.837523 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9zc\" (UniqueName: \"kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc\") pod \"maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:17.915577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:17.915543 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:18.026305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:18.026274 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" event={"ID":"fda4eb6a-85f1-428e-8193-33da6f1482e4","Type":"ContainerStarted","Data":"71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2"} Oct 30 05:45:18.026440 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:18.026326 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" event={"ID":"fda4eb6a-85f1-428e-8193-33da6f1482e4","Type":"ContainerStarted","Data":"a71a5c0ba3a64774570443671014654babf5bc985430a96838ad163ce57010c2"} Oct 30 05:45:18.026440 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:18.026367 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:18.071347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:18.070019 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" podStartSLOduration=2.069999717 podStartE2EDuration="2.069999717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:18.06943258 +0000 UTC m=+772.177471615" watchObservedRunningTime="2026-10-30 05:45:18.069999717 +0000 UTC m=+772.178038751" Oct 30 05:45:18.254399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:18.254375 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk"] Oct 30 05:45:18.256510 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:18.256481 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7fb19f_0864_4108_a51c_643f1d10fe16.slice/crio-c871c5c3a9bc7ccba9c9d8fb6ed59cbae9bd2d85e2041fe12add8a900d3e576e WatchSource:0}: Error finding container c871c5c3a9bc7ccba9c9d8fb6ed59cbae9bd2d85e2041fe12add8a900d3e576e: Status 404 returned error can't find the container with id c871c5c3a9bc7ccba9c9d8fb6ed59cbae9bd2d85e2041fe12add8a900d3e576e Oct 30 05:45:19.006458 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:19.006385 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 05:45:19.031315 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:19.031282 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" event={"ID":"9e7fb19f-0864-4108-a51c-643f1d10fe16","Type":"ContainerStarted","Data":"bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a"} Oct 30 05:45:19.031474 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:19.031343 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:19.031474 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:19.031360 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" event={"ID":"9e7fb19f-0864-4108-a51c-643f1d10fe16","Type":"ContainerStarted","Data":"c871c5c3a9bc7ccba9c9d8fb6ed59cbae9bd2d85e2041fe12add8a900d3e576e"} Oct 30 05:45:19.105027 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:19.104969 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" podStartSLOduration=1.735218202 podStartE2EDuration="2.10494805s" totalImagesPullingTime="369.729848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:19.102897662 +0000 UTC m=+773.210936707" watchObservedRunningTime="2026-10-30 05:45:19.10494805 +0000 UTC m=+773.212987082" Oct 30 05:45:21.016521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:21.016494 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 05:45:22.984577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:22.984212 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:45:22.984577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:22.984278 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:45:24.805801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.805766 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl"] Oct 30 05:45:24.813845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.813815 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.831975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.831943 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl"] Oct 30 05:45:24.846800 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.843353 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-1354a5-serving-cert\"" Oct 30 05:45:24.861292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.851216 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qprg\" (UniqueName: \"kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.861292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.851285 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.861292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.851361 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.861292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.851445 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.952250 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.952215 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qprg\" (UniqueName: \"kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.952424 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.952263 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.952424 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.952327 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.952424 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.952393 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.952585 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:45:24.952484 5980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-serving-cert: secret "maas-api-e2e-worker-w6-1354a5-serving-cert" not found Oct 30 05:45:24.952585 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:45:24.952579 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls podName:f1fc41cf-f675-4ada-b506-def7d09a8f41 nodeName:}" failed. No retries permitted until 2026-10-30 05:45:25.452554878 +0000 UTC m=+779.560593903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls") pod "maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" (UID: "f1fc41cf-f675-4ada-b506-def7d09a8f41") : secret "maas-api-e2e-worker-w6-1354a5-serving-cert" not found Oct 30 05:45:24.953061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.953040 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.955076 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.955050 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:24.983884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:24.981752 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qprg\" (UniqueName: \"kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:25.048089 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:25.048064 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 05:45:25.458611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:25.458581 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:25.461318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:25.461286 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\") pod \"maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:25.732224 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:25.732144 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:26.035807 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.035778 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl"] Oct 30 05:45:26.039850 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:45:26.039799 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc41cf_f675_4ada_b506_def7d09a8f41.slice/crio-6d51d2bf2aa7b2552c62d745e28a9b3dbec2e053779fdad4df636168331277ca WatchSource:0}: Error finding container 6d51d2bf2aa7b2552c62d745e28a9b3dbec2e053779fdad4df636168331277ca: Status 404 returned error can't find the container with id 6d51d2bf2aa7b2552c62d745e28a9b3dbec2e053779fdad4df636168331277ca Oct 30 05:45:26.078962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.078929 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" event={"ID":"f1fc41cf-f675-4ada-b506-def7d09a8f41","Type":"ContainerStarted","Data":"6d51d2bf2aa7b2552c62d745e28a9b3dbec2e053779fdad4df636168331277ca"} Oct 30 05:45:26.080638 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.080607 5980 generic.go:356] "Generic (PLEG): container finished" podID="66025822-a5de-4e85-8d69-b7fb524b5858" containerID="5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217" exitCode=0 Oct 30 05:45:26.080782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.080673 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" event={"ID":"66025822-a5de-4e85-8d69-b7fb524b5858","Type":"ContainerDied","Data":"5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217"} Oct 30 05:45:26.081128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.081092 5980 scope.go:122] "RemoveContainer" containerID="5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217" Oct 30 05:45:26.082470 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.082439 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:26.082584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.082506 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:26.082584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:26.082536 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:27.096980 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.096776 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" event={"ID":"f1fc41cf-f675-4ada-b506-def7d09a8f41","Type":"ContainerStarted","Data":"672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6"} Oct 30 05:45:27.096980 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.096943 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:27.099428 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.099404 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" event={"ID":"66025822-a5de-4e85-8d69-b7fb524b5858","Type":"ContainerStarted","Data":"d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba"} Oct 30 05:45:27.142076 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.141925 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" podStartSLOduration=2.860437128 podStartE2EDuration="3.141910601s" totalImagesPullingTime="281.473473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:27.141443626 +0000 UTC m=+781.249482660" watchObservedRunningTime="2026-10-30 05:45:27.141910601 +0000 UTC m=+781.249949633" Oct 30 05:45:27.221445 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.221402 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:27.221652 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.221623 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 05:45:27.221751 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:27.221678 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 05:45:28.029298 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:28.029273 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 05:45:28.222319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:28.222090 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 05:45:28.222319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:28.222190 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 05:45:29.111982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.111948 5980 generic.go:356] "Generic (PLEG): container finished" podID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerID="fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe" exitCode=0 Oct 30 05:45:29.112240 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.112017 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" event={"ID":"936d2db7-98ed-4b19-bf3c-d302b57cf15e","Type":"ContainerDied","Data":"fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe"} Oct 30 05:45:29.112516 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.112498 5980 scope.go:122] "RemoveContainer" containerID="fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe" Oct 30 05:45:29.114183 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.114150 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:29.114293 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.114236 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:29.114293 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.114283 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:29.222148 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.222092 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 05:45:29.222265 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:29.222175 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 05:45:30.044975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.044940 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 05:45:30.131725 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.131689 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" event={"ID":"936d2db7-98ed-4b19-bf3c-d302b57cf15e","Type":"ContainerStarted","Data":"c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f"} Oct 30 05:45:30.223135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.221745 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 05:45:30.223135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.221809 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 05:45:30.971596 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.971552 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:30.971815 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.971693 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:45:30.971815 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:30.971743 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:45:31.229195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:31.228544 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 05:45:31.229195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:31.228610 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 05:45:31.976134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:31.971502 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:45:31.976134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:31.971572 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:45:32.224134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:32.223574 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 05:45:32.224134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:32.223654 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 05:45:32.978135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:32.976584 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:45:32.978135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:32.976687 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:45:33.141134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:33.136367 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 05:45:33.255188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:33.253429 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:33.266132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:33.262198 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:33.306133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:33.305381 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 05:45:33.976133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:33.973979 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:45:33.976133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:33.974056 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:45:34.975138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:34.973485 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:45:34.975138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:34.973545 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:45:36.327037 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:36.325439 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:36.327037 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:36.326719 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:36.332540 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:36.332365 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 05:45:38.951991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:38.951949 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:45:38.952493 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:38.952017 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:45:38.992321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:38.992274 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:45:38.992514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:38.992366 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:45:41.454387 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.454353 5980 generic.go:356] "Generic (PLEG): container finished" podID="f8723f03-8296-4c76-b138-6375e3561d45" containerID="9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc" exitCode=0 Oct 30 05:45:41.454936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.454424 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" event={"ID":"f8723f03-8296-4c76-b138-6375e3561d45","Type":"ContainerDied","Data":"9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc"} Oct 30 05:45:41.454936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.454850 5980 scope.go:122] "RemoveContainer" containerID="9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc" Oct 30 05:45:41.456177 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.456157 5980 generic.go:356] "Generic (PLEG): container finished" podID="e2309786-9bb4-4274-962f-47f3697e5309" containerID="24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94" exitCode=0 Oct 30 05:45:41.456274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.456226 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" event={"ID":"e2309786-9bb4-4274-962f-47f3697e5309","Type":"ContainerDied","Data":"24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94"} Oct 30 05:45:41.456274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.456227 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:41.456347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.456293 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:41.456347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.456326 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:41.456525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.456512 5980 scope.go:122] "RemoveContainer" containerID="24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94" Oct 30 05:45:41.457703 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.457678 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:41.457785 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.457736 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:41.457831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:41.457782 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:42.470133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.467278 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" event={"ID":"f8723f03-8296-4c76-b138-6375e3561d45","Type":"ContainerStarted","Data":"a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b"} Oct 30 05:45:42.474131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.470671 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" event={"ID":"e2309786-9bb4-4274-962f-47f3697e5309","Type":"ContainerStarted","Data":"2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9"} Oct 30 05:45:42.696326 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.696267 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:42.697094 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.697067 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 05:45:42.697344 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.697310 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 05:45:42.832222 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.832183 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:42.832394 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.832299 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 05:45:42.832394 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:42.832350 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 05:45:43.138977 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:45:43.138885 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:45:43.695530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:43.695493 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 05:45:43.696151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:43.696092 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 05:45:43.832087 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:43.832049 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 05:45:43.832347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:43.832126 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 05:45:44.698525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:44.698489 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 05:45:44.699520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:44.699084 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 05:45:44.832301 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:44.832267 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 05:45:44.832466 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:44.832315 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 05:45:45.695528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:45.695489 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 05:45:45.695814 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:45.695785 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 05:45:45.832153 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:45.832098 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 05:45:45.832632 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:45.832173 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 05:45:46.695031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:46.694990 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 05:45:46.695390 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:46.695354 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 05:45:46.831714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:46.831660 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 05:45:46.831919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:46.831771 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 05:45:47.695679 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:47.695634 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 05:45:47.696268 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:47.695709 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 05:45:48.001552 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:48.001441 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:48.002047 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:48.001988 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:48.003436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:48.003399 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 05:45:48.699689 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:48.699660 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:48.700222 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:48.700055 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:45:48.700795 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:45:48.700776 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 05:46:02.189281 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.189243 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr"] Oct 30 05:46:02.192933 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.192911 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.225050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.225013 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-128436\"/\"kube-root-ca.crt\"" Oct 30 05:46:02.225906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.225876 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-128436\"/\"openshift-service-ca.crt\"" Oct 30 05:46:02.226048 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.226014 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-128436\"/\"default-dockercfg-4hqd7\"" Oct 30 05:46:02.226717 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.226694 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-128436\"/\"simulator-w3-128436-kserve-self-signed-certs\"" Oct 30 05:46:02.238942 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.238907 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr"] Oct 30 05:46:02.366783 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.366738 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.366962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.366795 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.366962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.366872 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.366962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.366914 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.367098 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.367074 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9mn\" (UniqueName: \"kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.367182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.367136 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468294 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468191 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468294 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468262 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468294 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468287 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468587 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468319 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468587 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468426 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9mn\" (UniqueName: \"kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468587 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468452 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468888 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468856 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.468978 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.468940 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.469237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.469188 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.470948 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.470920 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.471086 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.471070 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.495326 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.495293 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9mn\" (UniqueName: \"kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn\") pod \"simulator-w3-128436-kserve-7758f99944-xf6rr\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.543768 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.543732 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:02.853559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:02.853530 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr"] Oct 30 05:46:02.855655 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:46:02.855626 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70cbe530_2176_4450_881b_74bd0a41eb1a.slice/crio-f6ace4886d3a0d65baca85f80fecbbb446f0e12884b04026f4894fe3f3da8e51 WatchSource:0}: Error finding container f6ace4886d3a0d65baca85f80fecbbb446f0e12884b04026f4894fe3f3da8e51: Status 404 returned error can't find the container with id f6ace4886d3a0d65baca85f80fecbbb446f0e12884b04026f4894fe3f3da8e51 Oct 30 05:46:03.630209 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:03.630175 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" event={"ID":"70cbe530-2176-4450-881b-74bd0a41eb1a","Type":"ContainerStarted","Data":"0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b"} Oct 30 05:46:03.630650 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:03.630222 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" event={"ID":"70cbe530-2176-4450-881b-74bd0a41eb1a","Type":"ContainerStarted","Data":"f6ace4886d3a0d65baca85f80fecbbb446f0e12884b04026f4894fe3f3da8e51"} Oct 30 05:46:03.681397 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:03.681345 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podStartSLOduration=1.6813292180000001 podStartE2EDuration="1.681329218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:03.679884811 +0000 UTC m=+817.787923857" watchObservedRunningTime="2026-10-30 05:46:03.681329218 +0000 UTC m=+817.789368251" Oct 30 05:46:04.544279 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:04.544244 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:12.544616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:12.544574 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:12.558848 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:12.558814 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:12.695945 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:12.695913 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:46:13.092689 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:46:13.092651 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:46:15.225298 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.225266 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c"] Oct 30 05:46:15.228386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.228366 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.249472 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.249449 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c"] Oct 30 05:46:15.261004 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.260977 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-197e9b\"/\"openshift-service-ca.crt\"" Oct 30 05:46:15.261004 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.260992 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-197e9b\"/\"kube-root-ca.crt\"" Oct 30 05:46:15.261286 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.261046 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-197e9b\"/\"default-dockercfg-7rlx7\"" Oct 30 05:46:15.261286 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.261079 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-197e9b\"/\"premium-w0-197e9b-kserve-self-signed-certs\"" Oct 30 05:46:15.335196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.335160 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.335196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.335204 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.335427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.335230 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.335427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.335308 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.335427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.335365 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.335427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.335407 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nw4\" (UniqueName: \"kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.437502 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.437465 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.437748 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.437517 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nw4\" (UniqueName: \"kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.437748 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.437648 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.437748 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.437694 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.437748 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.437739 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.438070 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.437765 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.438282 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.438254 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.438559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.438532 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.439200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.439179 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.441248 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.441203 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.441610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.441564 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.464311 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.464284 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nw4\" (UniqueName: \"kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4\") pod \"premium-w0-197e9b-kserve-79984d9bcc-hwt6c\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.540189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.540083 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:15.851514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:15.851388 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c"] Oct 30 05:46:15.853803 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:46:15.853777 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f906dd9_e8c3_4be2_9079_3b1aa10970f9.slice/crio-58c4d179b5cea17abe6a9f6c0afba8f0702c025e18b726fbc53d97988e9edf66 WatchSource:0}: Error finding container 58c4d179b5cea17abe6a9f6c0afba8f0702c025e18b726fbc53d97988e9edf66: Status 404 returned error can't find the container with id 58c4d179b5cea17abe6a9f6c0afba8f0702c025e18b726fbc53d97988e9edf66 Oct 30 05:46:16.704535 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:16.704491 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" event={"ID":"9f906dd9-e8c3-4be2-9079-3b1aa10970f9","Type":"ContainerStarted","Data":"3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb"} Oct 30 05:46:16.704535 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:16.704540 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" event={"ID":"9f906dd9-e8c3-4be2-9079-3b1aa10970f9","Type":"ContainerStarted","Data":"58c4d179b5cea17abe6a9f6c0afba8f0702c025e18b726fbc53d97988e9edf66"} Oct 30 05:46:16.749811 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:16.749760 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podStartSLOduration=1.749746503 podStartE2EDuration="1.749746503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:16.74797632 +0000 UTC m=+830.856015350" watchObservedRunningTime="2026-10-30 05:46:16.749746503 +0000 UTC m=+830.857785536" Oct 30 05:46:17.540834 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:17.540795 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:19.746697 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.746649 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz"] Oct 30 05:46:19.750471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.750444 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.772230 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.772197 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4a68a1-openshift-default-dockercfg-mbxx7\"" Oct 30 05:46:19.782352 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.782322 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz"] Oct 30 05:46:19.791127 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791074 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791197 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rct6p\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791255 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791291 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791322 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791354 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791457 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791485 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.791627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.791544 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.892584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892539 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.892798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892589 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.892798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892650 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.892798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892697 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.892798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892736 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rct6p\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.892798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892779 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892817 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892854 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.892886 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893423 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.893342 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893423 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.893409 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893585 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.893480 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893706 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.893682 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.893919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.893889 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.895814 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.895785 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.896004 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.895987 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.918678 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.918629 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:19.920097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:19.920048 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rct6p\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:20.066207 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.065485 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:20.398322 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.398266 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz"] Oct 30 05:46:20.401007 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:46:20.400983 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96bb7f7_7077_4a32_9c20_0532cc7d6307.slice/crio-4e0f1ce6803de652e33c23e4ee0a6f5e8b9348388d96963b740bfaa57a3da02c WatchSource:0}: Error finding container 4e0f1ce6803de652e33c23e4ee0a6f5e8b9348388d96963b740bfaa57a3da02c: Status 404 returned error can't find the container with id 4e0f1ce6803de652e33c23e4ee0a6f5e8b9348388d96963b740bfaa57a3da02c Oct 30 05:46:20.403548 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.403514 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:20.403631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.403581 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:20.403631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.403610 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:20.729796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.729705 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" event={"ID":"f96bb7f7-7077-4a32-9c20-0532cc7d6307","Type":"ContainerStarted","Data":"731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575"} Oct 30 05:46:20.729796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.729772 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" event={"ID":"f96bb7f7-7077-4a32-9c20-0532cc7d6307","Type":"ContainerStarted","Data":"4e0f1ce6803de652e33c23e4ee0a6f5e8b9348388d96963b740bfaa57a3da02c"} Oct 30 05:46:20.810490 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:20.810418 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podStartSLOduration=1.81039863 podStartE2EDuration="1.81039863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:20.795446112 +0000 UTC m=+834.903485145" watchObservedRunningTime="2026-10-30 05:46:20.81039863 +0000 UTC m=+834.918437664" Oct 30 05:46:21.066647 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:21.066494 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:21.069040 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:21.068656 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:46:21.069040 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:21.068723 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 05:46:22.071141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:22.068525 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:46:22.071141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:22.068586 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 05:46:23.069134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:23.067676 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:46:23.069134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:23.067759 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 05:46:24.072563 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:24.072518 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:46:24.073195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:24.073162 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 05:46:25.066491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:25.066455 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:46:25.066763 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:25.066736 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 05:46:25.540376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:25.540344 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:25.552931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:25.552904 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:25.797757 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:25.797488 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:46:26.068009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:26.067509 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:46:26.068009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:26.067569 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 05:46:27.071203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:27.071171 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:27.071642 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:27.071453 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:27.072184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:27.072164 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 05:46:43.093731 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:46:43.093690 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:46:50.436794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.436753 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5"] Oct 30 05:46:50.440491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.440463 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.462956 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.462930 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e2e3d0-openshift-default-dockercfg-dkznd\"" Oct 30 05:46:50.474064 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.474036 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5"] Oct 30 05:46:50.557049 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557012 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557262 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557086 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557262 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557220 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557389 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557283 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557389 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557332 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557501 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557436 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557501 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557475 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557595 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557570 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6gl\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.557636 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.557593 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659091 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659048 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659091 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659099 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659155 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659225 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659262 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659324 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6gl\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659363 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659412 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659457 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659596 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659617 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659915 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659893 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.659986 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.659922 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.660045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.660027 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.661574 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.661550 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.662181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.662153 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.692887 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.692823 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.696724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.696703 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6gl\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:50.753671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:50.753628 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:51.122580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:51.122551 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5"] Oct 30 05:46:51.124099 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:46:51.124071 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaf7b10_aed3_4bfa_a836_b7a51d037f5f.slice/crio-6690468f79f9be5263fc84f7bfb6113dffc93ed3ca076f340d0fcce624707328 WatchSource:0}: Error finding container 6690468f79f9be5263fc84f7bfb6113dffc93ed3ca076f340d0fcce624707328: Status 404 returned error can't find the container with id 6690468f79f9be5263fc84f7bfb6113dffc93ed3ca076f340d0fcce624707328 Oct 30 05:46:51.126481 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:51.126447 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:51.126561 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:51.126514 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:51.126561 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:51.126543 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:51.949992 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:51.949956 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" event={"ID":"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f","Type":"ContainerStarted","Data":"b0a2feb369bd1e26df4cfb6a745822f0382115bd801f13477eb87729f2210149"} Oct 30 05:46:51.950596 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:51.950571 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" event={"ID":"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f","Type":"ContainerStarted","Data":"6690468f79f9be5263fc84f7bfb6113dffc93ed3ca076f340d0fcce624707328"} Oct 30 05:46:52.109639 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:52.109582 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podStartSLOduration=2.10956231 podStartE2EDuration="2.10956231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:52.09358263 +0000 UTC m=+866.201621704" watchObservedRunningTime="2026-10-30 05:46:52.10956231 +0000 UTC m=+866.217601345" Oct 30 05:46:52.754220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:52.754185 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:52.755773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:52.755747 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:46:52.755913 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:52.755801 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 05:46:53.754894 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:53.754862 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:46:53.755327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:53.754912 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 05:46:54.754651 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:54.754614 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:46:54.754860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:54.754670 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 05:46:55.754752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:55.754715 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:46:55.755266 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:55.754774 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 05:46:56.754643 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:56.754608 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:46:56.754850 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:56.754665 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 05:46:57.754572 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:57.754534 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:46:57.754746 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:57.754597 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 05:46:58.759022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:58.758995 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:58.759539 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:58.759440 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:46:58.760080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:46:58.760063 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 05:47:05.663685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.663652 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz"] Oct 30 05:47:05.668398 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.668372 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.696612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.696575 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a35835-openshift-default-dockercfg-rbcbq\"" Oct 30 05:47:05.726052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.726015 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz"] Oct 30 05:47:05.788814 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.788778 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.788982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.788843 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.788982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.788919 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.788982 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.788963 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.789172 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.789034 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.789172 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.789127 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.789280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.789182 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.789280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.789220 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.789379 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.789337 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db65t\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892200 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892265 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892301 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892388 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db65t\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892431 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892816 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892876 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892912 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.893131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.892954 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.896350 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.896299 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.896483 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.896374 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.896733 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.896550 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.896733 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.896691 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.897412 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.897386 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.897931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.897648 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.898052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.897879 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.935667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.935605 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.936802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.936776 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db65t\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:05.982979 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:05.982956 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:06.340559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:06.340533 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz"] Oct 30 05:47:06.342674 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:47:06.342645 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a4884_4a72_4f87_b0af_af22f3123dae.slice/crio-5193d51a13ebca5d157b0bb638a795cd213f09e297845de5f89a23c77773d3f7 WatchSource:0}: Error finding container 5193d51a13ebca5d157b0bb638a795cd213f09e297845de5f89a23c77773d3f7: Status 404 returned error can't find the container with id 5193d51a13ebca5d157b0bb638a795cd213f09e297845de5f89a23c77773d3f7 Oct 30 05:47:06.345709 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:06.345669 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:06.345805 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:06.345763 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:06.345862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:06.345807 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:07.103170 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:07.099909 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" event={"ID":"5a0a4884-4a72-4f87-b0af-af22f3123dae","Type":"ContainerStarted","Data":"9a377d083e767f798d9906d825d4afbd75dedc11cdacc5554222c92f591d38f3"} Oct 30 05:47:07.103170 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:07.100168 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" event={"ID":"5a0a4884-4a72-4f87-b0af-af22f3123dae","Type":"ContainerStarted","Data":"5193d51a13ebca5d157b0bb638a795cd213f09e297845de5f89a23c77773d3f7"} Oct 30 05:47:07.197134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:07.197001 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podStartSLOduration=2.196985171 podStartE2EDuration="2.196985171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:07.196504137 +0000 UTC m=+881.304543170" watchObservedRunningTime="2026-10-30 05:47:07.196985171 +0000 UTC m=+881.305024211" Oct 30 05:47:07.984458 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:07.984427 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:07.988237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:07.988207 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 05:47:07.988460 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:07.988431 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 05:47:08.987134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:08.986698 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 05:47:08.987134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:08.986761 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 05:47:09.983517 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:09.983478 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 05:47:09.983791 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:09.983759 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 05:47:10.988133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:10.987482 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 05:47:10.988133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:10.987546 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 05:47:11.984559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:11.984524 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 05:47:11.984839 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:11.984812 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 05:47:13.094896 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:47:13.094853 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:47:13.336009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:13.321032 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:13.336009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:13.322232 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:13.343612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:13.343516 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 05:47:41.344319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:41.344240 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:47:41.344319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:41.344306 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:43.086591 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:47:43.086559 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:47:47.491050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.491019 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t"] Oct 30 05:47:47.495058 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.495035 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.526164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.526139 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-128436\"/\"distinct-w3-128436-kserve-self-signed-certs\"" Oct 30 05:47:47.529151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.529128 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t"] Oct 30 05:47:47.590998 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.590958 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.592392 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.591055 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfwb\" (UniqueName: \"kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.592392 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.591139 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.592392 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.591180 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.592392 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.591253 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.592392 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.591291 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.692585 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.692549 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfwb\" (UniqueName: \"kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.692778 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.692607 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.692778 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.692648 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.692868 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.692826 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.692868 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.692860 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.692975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.692949 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.693102 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.693071 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.693204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.693183 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.693304 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.693282 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.695288 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.695258 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.695575 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.695556 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.723047 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.723018 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfwb\" (UniqueName: \"kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb\") pod \"distinct-w3-128436-kserve-6c75565476-xjq6t\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:47.808273 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:47.808196 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:48.116322 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:48.116302 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t"] Oct 30 05:47:48.118515 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:47:48.118492 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0f49717_ed4b_4dd0_ad8f_8f1ffb40f21f.slice/crio-9e6889001ee562d1369ef588dc43f401da14e2ffba5ba048ca1db5092d09a1aa WatchSource:0}: Error finding container 9e6889001ee562d1369ef588dc43f401da14e2ffba5ba048ca1db5092d09a1aa: Status 404 returned error can't find the container with id 9e6889001ee562d1369ef588dc43f401da14e2ffba5ba048ca1db5092d09a1aa Oct 30 05:47:48.362382 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:48.362348 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" event={"ID":"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f","Type":"ContainerStarted","Data":"cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459"} Oct 30 05:47:48.362382 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:48.362387 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" event={"ID":"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f","Type":"ContainerStarted","Data":"9e6889001ee562d1369ef588dc43f401da14e2ffba5ba048ca1db5092d09a1aa"} Oct 30 05:47:48.413275 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:48.413189 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podStartSLOduration=1.413175182 podStartE2EDuration="1.413175182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:48.411207704 +0000 UTC m=+922.519246737" watchObservedRunningTime="2026-10-30 05:47:48.413175182 +0000 UTC m=+922.521214231" Oct 30 05:47:48.808713 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:48.808632 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:57.809167 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:57.809137 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:57.822882 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:57.822858 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:47:58.452589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:47:58.452560 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:48:02.053929 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.053897 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk"] Oct 30 05:48:02.058320 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.058301 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.094677 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.094644 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk"] Oct 30 05:48:02.108080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.107825 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.107929 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.107993 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.108036 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.108091 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.108168 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.108190 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.108284 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrm9h\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.108436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.108340 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.208872 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.208838 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.208882 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.208916 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.208953 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.208972 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209036 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrm9h\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209080 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209160 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209232 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209283 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209381 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209423 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209480 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.209864 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.209846 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.211680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.211656 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.211838 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.211821 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.233959 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.233934 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.234818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.234795 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrm9h\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h\") pod \"maas-default-gateway-openshift-default-58847f9fc5-dvbqk\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.371471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.371439 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:02.694448 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.694423 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk"] Oct 30 05:48:02.696962 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:48:02.696936 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40be304_8a11_48bd_aa34_94577f784fb1.slice/crio-b53c4e0ca0890aff8753d14f4dd544351d982314de81a30cb1af536f370834ec WatchSource:0}: Error finding container b53c4e0ca0890aff8753d14f4dd544351d982314de81a30cb1af536f370834ec: Status 404 returned error can't find the container with id b53c4e0ca0890aff8753d14f4dd544351d982314de81a30cb1af536f370834ec Oct 30 05:48:02.700009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.699981 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:02.700084 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.700047 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:02.700084 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:02.700074 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:03.493588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:03.493552 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" event={"ID":"d40be304-8a11-48bd-aa34-94577f784fb1","Type":"ContainerStarted","Data":"a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c"} Oct 30 05:48:03.494041 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:03.493621 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" event={"ID":"d40be304-8a11-48bd-aa34-94577f784fb1","Type":"ContainerStarted","Data":"b53c4e0ca0890aff8753d14f4dd544351d982314de81a30cb1af536f370834ec"} Oct 30 05:48:03.566130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:03.562537 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podStartSLOduration=1.562518415 podStartE2EDuration="1.562518415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:03.560326387 +0000 UTC m=+937.668365420" watchObservedRunningTime="2026-10-30 05:48:03.562518415 +0000 UTC m=+937.670557451" Oct 30 05:48:04.377131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:04.373437 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:04.382128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:04.380567 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 05:48:04.382128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:04.380621 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 05:48:05.086420 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.086384 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn"] Oct 30 05:48:05.097836 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.097808 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.114685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.114663 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn"] Oct 30 05:48:05.140626 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140599 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.140796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140659 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.140796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140747 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.140796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140772 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.140898 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140849 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.140931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140898 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.140976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.140922 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftsc\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.141070 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.141054 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.141141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.141123 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242089 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242175 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242200 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftsc\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242257 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242305 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242410 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242446 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242464 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242508 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242526 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242805 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242779 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.242891 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.242871 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.243213 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.243191 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.243297 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.243228 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.244580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.244563 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.244801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.244784 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.296158 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.296099 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.297816 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.297766 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftsc\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.381395 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.372488 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 05:48:05.381395 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.380856 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 05:48:05.420637 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.420607 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:05.801470 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.797641 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn"] Oct 30 05:48:05.807203 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:48:05.806063 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ccf1dc3_743b_41af_9171_1a836d2f839e.slice/crio-7c70ffbb52b19e38a2ec3aa2cb6d7929ac70d998a5cdcd348d09438017d0bc8d WatchSource:0}: Error finding container 7c70ffbb52b19e38a2ec3aa2cb6d7929ac70d998a5cdcd348d09438017d0bc8d: Status 404 returned error can't find the container with id 7c70ffbb52b19e38a2ec3aa2cb6d7929ac70d998a5cdcd348d09438017d0bc8d Oct 30 05:48:05.809919 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.809880 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:05.810009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.809952 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:05.810009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:05.809983 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:06.380484 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.380449 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 05:48:06.381068 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.381037 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 05:48:06.512924 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.512890 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" event={"ID":"6ccf1dc3-743b-41af-9171-1a836d2f839e","Type":"ContainerStarted","Data":"114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4"} Oct 30 05:48:06.513134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.512939 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" event={"ID":"6ccf1dc3-743b-41af-9171-1a836d2f839e","Type":"ContainerStarted","Data":"7c70ffbb52b19e38a2ec3aa2cb6d7929ac70d998a5cdcd348d09438017d0bc8d"} Oct 30 05:48:06.751030 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.750940 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podStartSLOduration=1.750924041 podStartE2EDuration="1.750924041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:06.591363023 +0000 UTC m=+940.699402055" watchObservedRunningTime="2026-10-30 05:48:06.750924041 +0000 UTC m=+940.858963072" Oct 30 05:48:06.752578 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.752558 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl"] Oct 30 05:48:06.758469 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.758449 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:06.783469 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.783441 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"openshift-service-ca.crt\"" Oct 30 05:48:06.784378 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.784357 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"distinct-w1-4a68a1-kserve-self-signed-certs\"" Oct 30 05:48:06.784612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.784598 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"default-dockercfg-576t9\"" Oct 30 05:48:06.784806 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.784792 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"kube-root-ca.crt\"" Oct 30 05:48:06.809135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.786214 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl"] Oct 30 05:48:06.888133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.887881 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:06.888133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.887956 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpm58\" (UniqueName: \"kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:06.888133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.887997 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:06.888133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.888022 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:06.888133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.888052 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:06.888133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:06.888091 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020050 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020154 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpm58\" (UniqueName: \"kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020192 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020216 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020243 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020283 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.020815 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.021040 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.030178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.024080 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.055284 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.055249 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpm58\" (UniqueName: \"kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.055549 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.055528 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.056774 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.056749 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs\") pod \"distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.144323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.144291 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:07.417151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.414308 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 05:48:07.417151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.414393 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 05:48:07.434131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.430189 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:07.434131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.430910 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 05:48:07.434131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.430964 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 05:48:07.537318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:07.537294 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl"] Oct 30 05:48:07.539912 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:48:07.539711 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb495806_9f5c_47d5_b035_f6b546928092.slice/crio-32b116714931fb20f87e24561a734e78b4afc4519e2fb2c8633eba684ce205d5 WatchSource:0}: Error finding container 32b116714931fb20f87e24561a734e78b4afc4519e2fb2c8633eba684ce205d5: Status 404 returned error can't find the container with id 32b116714931fb20f87e24561a734e78b4afc4519e2fb2c8633eba684ce205d5 Oct 30 05:48:08.386133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.377486 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 05:48:08.386133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.377560 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 05:48:08.421059 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.421019 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 05:48:08.421590 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.421077 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 05:48:08.544082 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.544042 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" event={"ID":"bb495806-9f5c-47d5-b035-f6b546928092","Type":"ContainerStarted","Data":"297a0d17cc057bf1ce303380039aedc895024c3a9a0d8bf4739c954859f93634"} Oct 30 05:48:08.544279 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.544093 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" event={"ID":"bb495806-9f5c-47d5-b035-f6b546928092","Type":"ContainerStarted","Data":"32b116714931fb20f87e24561a734e78b4afc4519e2fb2c8633eba684ce205d5"} Oct 30 05:48:08.593134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:08.593029 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podStartSLOduration=2.593011804 podStartE2EDuration="2.593011804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:08.589562229 +0000 UTC m=+942.697601262" watchObservedRunningTime="2026-10-30 05:48:08.593011804 +0000 UTC m=+942.701050853" Oct 30 05:48:09.145203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:09.145170 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:09.378207 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:09.378173 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 05:48:09.378510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:09.378469 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 05:48:09.421805 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:09.421725 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 05:48:09.421805 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:09.421783 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 05:48:10.421660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:10.421616 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 05:48:10.421840 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:10.421679 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 05:48:11.279140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:11.277465 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:11.279140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:11.278614 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:11.288334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:11.288310 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 05:48:11.422579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:11.422548 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 05:48:11.423060 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:11.422619 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 05:48:12.421622 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:12.421583 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 05:48:12.421803 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:12.421644 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 05:48:13.086857 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:48:13.086817 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:48:13.425641 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:13.425543 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:13.425846 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:13.425822 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:13.426419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:13.426403 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 05:48:17.144716 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:17.144680 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:17.158426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:17.158393 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:17.625237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:17.625206 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:48:21.161526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.161495 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd"] Oct 30 05:48:21.165550 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.165531 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.208880 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.208847 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd"] Oct 30 05:48:21.244176 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244152 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244176 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244183 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244356 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244332 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244392 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244368 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krw6k\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244440 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244407 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244489 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244574 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244557 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244638 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244621 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.244735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.244723 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.345913 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.345883 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.345928 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krw6k\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.345978 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346018 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346047 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346333 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346096 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346333 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346176 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346333 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346221 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346333 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346245 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346508 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346451 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346566 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346543 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346708 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.346869 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.346847 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.347078 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.347057 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.348714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.348690 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.348831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.348812 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.370449 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.370428 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.371143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.371122 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krw6k\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.478541 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.478476 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:21.814923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.814817 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd"] Oct 30 05:48:21.817098 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:48:21.817071 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ba01ac_dfc1_43f0_8550_d101eeddb858.slice/crio-bf6a1e4e1019ee3dc2f109ae7d73e4d217972c79901240b51ef7e0cd6ab70d47 WatchSource:0}: Error finding container bf6a1e4e1019ee3dc2f109ae7d73e4d217972c79901240b51ef7e0cd6ab70d47: Status 404 returned error can't find the container with id bf6a1e4e1019ee3dc2f109ae7d73e4d217972c79901240b51ef7e0cd6ab70d47 Oct 30 05:48:21.819700 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.819669 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:21.819782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.819734 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:21.819782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:21.819763 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:22.693693 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:22.693649 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" event={"ID":"85ba01ac-dfc1-43f0-8550-d101eeddb858","Type":"ContainerStarted","Data":"921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c"} Oct 30 05:48:22.694130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:22.693711 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" event={"ID":"85ba01ac-dfc1-43f0-8550-d101eeddb858","Type":"ContainerStarted","Data":"bf6a1e4e1019ee3dc2f109ae7d73e4d217972c79901240b51ef7e0cd6ab70d47"} Oct 30 05:48:23.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:23.487511 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:23.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:23.489097 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:48:23.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:23.489162 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:48:24.483134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:24.481491 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:48:24.483134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:24.481553 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:48:25.483134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:25.482541 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:48:25.483134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:25.482617 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:48:26.493144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:26.487588 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:48:26.493144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:26.487649 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:48:27.480625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:27.480581 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:48:27.480846 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:27.480645 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:48:28.518053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.517989 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podStartSLOduration=7.517970822 podStartE2EDuration="7.517970822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:22.7813962 +0000 UTC m=+956.889435233" watchObservedRunningTime="2026-10-30 05:48:28.517970822 +0000 UTC m=+962.626009855" Oct 30 05:48:28.519393 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.519366 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf"] Oct 30 05:48:28.524778 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.524757 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.544591 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.544562 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf"] Oct 30 05:48:28.552542 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.552513 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.552704 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.552555 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.552704 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.552656 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjxl\" (UniqueName: \"kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.552835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.552701 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.552835 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.552766 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.552957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.552870 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.555033 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.555008 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-128436\"/\"distinct-2-w3-128436-kserve-self-signed-certs\"" Oct 30 05:48:28.604100 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.604068 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:28.604640 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.604619 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:28.605833 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.605805 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:48:28.653660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.653633 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.653859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.653691 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.653859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.653713 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.653859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.653797 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjxl\" (UniqueName: \"kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.653859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.653827 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.654058 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.653910 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.654249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.654225 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.654456 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.654433 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.654618 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.654565 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.658140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.656966 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.658140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.657412 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.683534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.683507 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjxl\" (UniqueName: \"kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl\") pod \"distinct-2-w3-128436-kserve-5f97768687-wqdrf\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:28.844634 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:28.844345 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:29.180225 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:29.180204 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf"] Oct 30 05:48:29.182925 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:48:29.182898 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c48ddd4_40f4_41ed_9350_5e24c07ef671.slice/crio-18ae16684f376aabaa4ad693345f7022cbde23f4daae571f31a49daf678b6696 WatchSource:0}: Error finding container 18ae16684f376aabaa4ad693345f7022cbde23f4daae571f31a49daf678b6696: Status 404 returned error can't find the container with id 18ae16684f376aabaa4ad693345f7022cbde23f4daae571f31a49daf678b6696 Oct 30 05:48:29.745568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:29.745528 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" event={"ID":"1c48ddd4-40f4-41ed-9350-5e24c07ef671","Type":"ContainerStarted","Data":"e4fabcd1e0192d61002fc526b9962a34ffb6faaa21af1bc4be6f5976cec7ce8b"} Oct 30 05:48:29.745568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:29.745574 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" event={"ID":"1c48ddd4-40f4-41ed-9350-5e24c07ef671","Type":"ContainerStarted","Data":"18ae16684f376aabaa4ad693345f7022cbde23f4daae571f31a49daf678b6696"} Oct 30 05:48:29.845327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:29.845295 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:29.879080 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:29.879018 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podStartSLOduration=1.878999817 podStartE2EDuration="1.878999817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:29.87497632 +0000 UTC m=+963.983015351" watchObservedRunningTime="2026-10-30 05:48:29.878999817 +0000 UTC m=+963.987038850" Oct 30 05:48:38.846575 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:38.845541 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:38.888131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:38.872395 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:39.840944 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:39.840916 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:48:43.089799 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:48:43.089759 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:48:52.354865 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.354831 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb"] Oct 30 05:48:52.359217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.359196 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.378942 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.378911 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb"] Oct 30 05:48:52.390121 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.390071 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"kube-root-ca.crt\"" Oct 30 05:48:52.390280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.390153 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"distinct-2-w5-a35835-kserve-self-signed-certs\"" Oct 30 05:48:52.390719 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.390694 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.390777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.390752 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.390936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.390916 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.391082 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.391064 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.391144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.391099 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.391236 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.391220 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.391727 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.391701 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"openshift-service-ca.crt\"" Oct 30 05:48:52.391940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.391924 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"default-dockercfg-644ms\"" Oct 30 05:48:52.492652 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.492606 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.492845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.492680 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.492845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.492765 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.492947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.492886 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.492947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.492911 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.493045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.492973 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.493246 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.493191 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.493246 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.493221 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.493616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.493372 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.495341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.495319 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.495531 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.495509 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.517651 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.517623 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5\") pod \"distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.672363 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.672273 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:48:52.983025 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:52.982901 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb"] Oct 30 05:48:52.993394 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:48:52.993309 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1605026c_45f3_4dfc_a9a5_6a9ebf5ce342.slice/crio-dd988a6fa6cb5d0b61a201f2168c7642df397a702e3d6f0e66bf888b46db2a9a WatchSource:0}: Error finding container dd988a6fa6cb5d0b61a201f2168c7642df397a702e3d6f0e66bf888b46db2a9a: Status 404 returned error can't find the container with id dd988a6fa6cb5d0b61a201f2168c7642df397a702e3d6f0e66bf888b46db2a9a Oct 30 05:48:53.912061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:53.912021 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" event={"ID":"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342","Type":"ContainerStarted","Data":"d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213"} Oct 30 05:48:53.912547 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:53.912068 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" event={"ID":"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342","Type":"ContainerStarted","Data":"dd988a6fa6cb5d0b61a201f2168c7642df397a702e3d6f0e66bf888b46db2a9a"} Oct 30 05:48:53.957661 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:53.957600 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podStartSLOduration=1.957588127 podStartE2EDuration="1.957588127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:53.955282973 +0000 UTC m=+988.063322006" watchObservedRunningTime="2026-10-30 05:48:53.957588127 +0000 UTC m=+988.065627158" Oct 30 05:48:54.673311 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:48:54.673276 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:49:02.672404 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:02.672368 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:49:02.685840 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:02.685810 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:49:02.976172 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:02.976064 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:49:06.736019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.735939 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn"] Oct 30 05:49:06.741714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.741694 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.762950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.762916 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn"] Oct 30 05:49:06.770147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.769681 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-197e9b\"/\"distinct-w0-197e9b-kserve-self-signed-certs\"" Oct 30 05:49:06.797486 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.797435 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.797657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.797513 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.799272 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.799236 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.799568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.799544 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.799780 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.799761 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvcl\" (UniqueName: \"kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.800001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.799983 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.905941 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.905901 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.906238 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.906211 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.906342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.906250 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.906342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.906334 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.906414 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.906401 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvcl\" (UniqueName: \"kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.906515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.906501 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.906974 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.906953 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.907813 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.907789 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.908088 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.908067 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.912787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.912743 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.914228 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.914203 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:06.937129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:06.935626 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvcl\" (UniqueName: \"kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl\") pod \"distinct-w0-197e9b-kserve-756cfdb47b-qhfpn\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:07.056002 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:07.055911 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:07.391016 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:07.390989 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn"] Oct 30 05:49:07.393914 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:49:07.393882 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod249063da_915b_4eaa_a85f_643d37c75157.slice/crio-57eeee0f3a31839522876b76d7720a839988edeabf413187c989c58be8e9e533 WatchSource:0}: Error finding container 57eeee0f3a31839522876b76d7720a839988edeabf413187c989c58be8e9e533: Status 404 returned error can't find the container with id 57eeee0f3a31839522876b76d7720a839988edeabf413187c989c58be8e9e533 Oct 30 05:49:08.014586 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:08.014551 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" event={"ID":"249063da-915b-4eaa-a85f-643d37c75157","Type":"ContainerStarted","Data":"cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b"} Oct 30 05:49:08.015008 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:08.014591 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" event={"ID":"249063da-915b-4eaa-a85f-643d37c75157","Type":"ContainerStarted","Data":"57eeee0f3a31839522876b76d7720a839988edeabf413187c989c58be8e9e533"} Oct 30 05:49:08.056138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:08.056044 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:13.095154 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:49:13.093454 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:49:17.056514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:17.056477 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:17.068625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:17.068592 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:17.109305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:17.109271 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:49:17.121693 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:17.121641 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podStartSLOduration=11.121626226 podStartE2EDuration="11.121626226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:08.072668193 +0000 UTC m=+1002.180707225" watchObservedRunningTime="2026-10-30 05:49:17.121626226 +0000 UTC m=+1011.229665257" Oct 30 05:49:29.774980 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.774914 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns"] Oct 30 05:49:29.781485 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.781461 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.801384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.801360 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"openshift-service-ca.crt\"" Oct 30 05:49:29.802745 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.801419 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns"] Oct 30 05:49:29.802859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.801556 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"kube-root-ca.crt\"" Oct 30 05:49:29.810553 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.810531 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"distinct-w4-e2e3d0-kserve-self-signed-certs\"" Oct 30 05:49:29.811442 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.811412 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"default-dockercfg-l9xmp\"" Oct 30 05:49:29.874249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.874223 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.874383 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.874333 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.874444 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.874427 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.874514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.874498 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.874552 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.874527 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlwc\" (UniqueName: \"kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.874604 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.874564 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.976611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.976523 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.976863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.976720 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.976863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.976748 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlwc\" (UniqueName: \"kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.977029 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.976876 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.977314 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.977025 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.977314 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.977152 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.978568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.978258 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.978568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.978522 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:29.994128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:29.979307 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:30.004128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:30.002493 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:30.004128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:30.002875 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:30.019969 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:30.019941 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlwc\" (UniqueName: \"kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc\") pod \"distinct-w4-e2e3d0-kserve-7d86656666-m5wns\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:30.107633 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:30.107589 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:30.415534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:30.415490 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns"] Oct 30 05:49:30.418035 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:49:30.418011 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42050220_36e0_4c55_a2e5_b4f1c66bebdf.slice/crio-75ad97176144c640cf4904049de10347d34e774a10497cf50db7393f83656ce2 WatchSource:0}: Error finding container 75ad97176144c640cf4904049de10347d34e774a10497cf50db7393f83656ce2: Status 404 returned error can't find the container with id 75ad97176144c640cf4904049de10347d34e774a10497cf50db7393f83656ce2 Oct 30 05:49:31.368718 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:31.368674 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" event={"ID":"42050220-36e0-4c55-a2e5-b4f1c66bebdf","Type":"ContainerStarted","Data":"d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a"} Oct 30 05:49:31.368718 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:31.368732 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" event={"ID":"42050220-36e0-4c55-a2e5-b4f1c66bebdf","Type":"ContainerStarted","Data":"75ad97176144c640cf4904049de10347d34e774a10497cf50db7393f83656ce2"} Oct 30 05:49:31.417524 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:31.415740 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podStartSLOduration=2.415713429 podStartE2EDuration="2.415713429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:31.413230337 +0000 UTC m=+1025.521269368" watchObservedRunningTime="2026-10-30 05:49:31.415713429 +0000 UTC m=+1025.523752460" Oct 30 05:49:32.108946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:32.108913 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:35.354775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.354744 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn"] Oct 30 05:49:35.359147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.359089 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.385955 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.385867 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn"] Oct 30 05:49:35.443651 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.443617 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.443887 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.443861 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.443960 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.443925 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.444062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.444031 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswxs\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.444175 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.444154 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.444279 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.444264 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.444336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.444302 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.444389 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.444369 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.444439 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.444397 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546039 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546002 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546257 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546053 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546257 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546102 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546257 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546168 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswxs\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546257 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546217 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546296 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546330 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546394 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546473 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546426 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546702 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546677 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546773 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546796 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.546862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.546850 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.547101 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.547081 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.548588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.548567 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.549399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.549377 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.570970 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.570950 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.571064 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.571014 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswxs\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.673321 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.673236 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:35.997409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:35.997384 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn"] Oct 30 05:49:35.998656 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:49:35.998631 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e514947_f603_4419_a683_1f36f2ff618b.slice/crio-a84cf059d9b1d794d2e6883c8fbc25649312217976df85fc3e3c0b03e09eeab6 WatchSource:0}: Error finding container a84cf059d9b1d794d2e6883c8fbc25649312217976df85fc3e3c0b03e09eeab6: Status 404 returned error can't find the container with id a84cf059d9b1d794d2e6883c8fbc25649312217976df85fc3e3c0b03e09eeab6 Oct 30 05:49:36.001472 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.001433 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.001567 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.001524 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.001567 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.001558 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.054193 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.054171 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz"] Oct 30 05:49:36.065973 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.065944 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.083460 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.083432 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz"] Oct 30 05:49:36.154296 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154249 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154296 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154298 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154489 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154422 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154493 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74cx\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154561 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154614 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154680 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154727 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154706 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.154773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.154760 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.255937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.255849 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.255937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.255899 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.255939 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256001 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74cx\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256070 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256139 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256210 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256234 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256293 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256499 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256641 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.256838 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.256807 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.257097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.257075 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.257205 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.257173 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.259227 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.259206 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.259227 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.259218 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.280549 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.280526 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.281076 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.281057 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74cx\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.363624 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.363583 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7"] Oct 30 05:49:36.369173 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.369146 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.391786 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.391563 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7"] Oct 30 05:49:36.393273 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.393055 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:36.423592 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.423544 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" event={"ID":"3e514947-f603-4419-a683-1f36f2ff618b","Type":"ContainerStarted","Data":"72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71"} Oct 30 05:49:36.423729 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.423629 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" event={"ID":"3e514947-f603-4419-a683-1f36f2ff618b","Type":"ContainerStarted","Data":"a84cf059d9b1d794d2e6883c8fbc25649312217976df85fc3e3c0b03e09eeab6"} Oct 30 05:49:36.459901 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.459877 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460268 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460242 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460317 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460348 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7sf4\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460412 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460438 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460466 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460491 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.460719 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.460594 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.476551 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.476499 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podStartSLOduration=1.4764787 podStartE2EDuration="1.4764787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:36.471238063 +0000 UTC m=+1030.579277120" watchObservedRunningTime="2026-10-30 05:49:36.4764787 +0000 UTC m=+1030.584517733" Oct 30 05:49:36.561844 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.561761 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.561972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.561854 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.561972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.561907 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.561972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.561936 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.561972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.561954 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7sf4\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.562154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.561992 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.562154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.562007 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.562154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.562023 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.562154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.562036 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.565573 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.562304 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.565573 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.563073 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.565573 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.563340 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.594162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.585280 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.594162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.593924 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.598488 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.598453 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7sf4\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.598645 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.598505 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.599986 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.599961 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.600938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.600888 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.674299 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.674211 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:36.689153 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.687304 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:36.751569 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.750971 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz"] Oct 30 05:49:36.786247 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:49:36.785529 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38cd2fb_3ec8_4bb3_9fc4_8fd06f4eabd4.slice/crio-fddc8784682c2b748ccdcf0873ffd680a1092ed7d76f13b143c50b49aa2c7313 WatchSource:0}: Error finding container fddc8784682c2b748ccdcf0873ffd680a1092ed7d76f13b143c50b49aa2c7313: Status 404 returned error can't find the container with id fddc8784682c2b748ccdcf0873ffd680a1092ed7d76f13b143c50b49aa2c7313 Oct 30 05:49:36.791182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.790380 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.791182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.790472 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.791182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:36.790512 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.003869 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.002755 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7"] Oct 30 05:49:37.006082 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:49:37.006050 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd53c91_9a4c_4243_a657_ce77ddd88282.slice/crio-cbd7810fb27e8d12d9aa7f978e60bb911158b7dd3e53289b01497875c5fcc3b3 WatchSource:0}: Error finding container cbd7810fb27e8d12d9aa7f978e60bb911158b7dd3e53289b01497875c5fcc3b3: Status 404 returned error can't find the container with id cbd7810fb27e8d12d9aa7f978e60bb911158b7dd3e53289b01497875c5fcc3b3 Oct 30 05:49:37.009369 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.008825 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.009369 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.008886 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.009369 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.008914 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.434875 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.434768 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" event={"ID":"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4","Type":"ContainerStarted","Data":"8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae"} Oct 30 05:49:37.434875 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.434841 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" event={"ID":"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4","Type":"ContainerStarted","Data":"fddc8784682c2b748ccdcf0873ffd680a1092ed7d76f13b143c50b49aa2c7313"} Oct 30 05:49:37.438784 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.438759 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" event={"ID":"afd53c91-9a4c-4243-a657-ce77ddd88282","Type":"ContainerStarted","Data":"1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc"} Oct 30 05:49:37.438984 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.438968 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" event={"ID":"afd53c91-9a4c-4243-a657-ce77ddd88282","Type":"ContainerStarted","Data":"cbd7810fb27e8d12d9aa7f978e60bb911158b7dd3e53289b01497875c5fcc3b3"} Oct 30 05:49:37.556694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.556634 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podStartSLOduration=1.5566135239999999 podStartE2EDuration="1.556613524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:37.502298355 +0000 UTC m=+1031.610337412" watchObservedRunningTime="2026-10-30 05:49:37.556613524 +0000 UTC m=+1031.664652573" Oct 30 05:49:37.556971 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.556939 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podStartSLOduration=1.556926406 podStartE2EDuration="1.556926406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:37.553379846 +0000 UTC m=+1031.661418904" watchObservedRunningTime="2026-10-30 05:49:37.556926406 +0000 UTC m=+1031.664965439" Oct 30 05:49:37.677137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.675681 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 05:49:37.677137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.675747 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 05:49:37.693192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:37.692241 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:38.395132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.394747 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:38.395132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.394868 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.395132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.394909 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 05:49:38.847129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.843944 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.847129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.844004 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 05:49:38.847129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.846051 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.847129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:38.846089 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:49:39.430132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.410749 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.430132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.410813 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 05:49:39.513146 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.502239 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.635242 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.635189 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.702194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.699264 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.702194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.699335 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 05:49:39.702194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.700058 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.702194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.700131 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:49:39.708307 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.708263 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.851134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.848219 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.851134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:39.848618 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:49:40.117141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.113052 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:40.154015 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.153976 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:40.395015 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.394512 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.395015 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.394577 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 05:49:40.666502 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.666394 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:49:40.684432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.675464 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.684432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.675520 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 05:49:40.697132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.691592 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.697132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:40.691647 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:49:41.429133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:41.429065 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:49:41.429601 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:41.429145 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 05:49:41.727697 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:41.727608 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 05:49:41.727958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:41.727925 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 05:49:41.728577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:41.728555 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:49:41.728780 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:41.728757 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:49:42.400522 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:42.400485 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.400782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:42.400753 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 05:49:42.677187 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:42.673678 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.677187 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:42.673733 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 05:49:42.692511 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:42.692403 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.692511 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:42.692457 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:49:43.133630 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:49:43.133595 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:49:43.395589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:43.395485 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 05:49:43.395589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:43.395568 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 05:49:43.690126 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:43.690018 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:49:43.691327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:43.691293 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:49:44.144829 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:44.144797 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:44.145529 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:44.145503 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:44.146413 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:44.146393 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:49:44.747548 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:44.747511 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:44.747975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:44.747932 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:44.748886 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:44.748865 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 05:49:45.117836 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:45.117803 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:45.118243 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:45.118221 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:45.119162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:45.119141 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 05:49:50.587129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.586431 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2"] Oct 30 05:49:50.621416 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.621387 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.668144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.652924 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2"] Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.719640 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.719702 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.719758 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.719868 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.719897 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxrd\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.719961 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.720023 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.720047 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.720969 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.720099 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871102 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871166 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxrd\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871254 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871323 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871349 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871405 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871466 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871514 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.871579 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.877520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.872370 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.882128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.881986 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.882270 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.882257 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.896129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.884648 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.936202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.926783 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.936426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.936408 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w"] Oct 30 05:49:50.949658 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.949629 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:50.971134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.971091 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.986651 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.986621 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.986865 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.986837 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxrd\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.987983 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.987957 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:50.991925 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:50.991902 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w"] Oct 30 05:49:51.026063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.026037 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:51.095096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.094880 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.094917 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095256 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095154 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095256 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095185 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095256 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095211 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8b5c\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095400 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095335 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095400 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095358 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095400 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095384 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.095532 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.095416 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.120595 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.120559 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn"] Oct 30 05:49:51.126406 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.125792 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.153729 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.153706 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn"] Oct 30 05:49:51.199953 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.199919 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.200219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200199 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.200339 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200319 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8b5c\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.200469 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200453 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.200607 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200592 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.200709 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200697 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.200852 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200838 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.200944 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.200931 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9zl\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.201066 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201052 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.201180 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201167 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.201280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201268 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.201384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201371 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.201476 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201464 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.201683 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201667 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.201804 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201791 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.201904 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.201890 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.202035 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.202022 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.202194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.202180 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.203717 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.203690 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.204467 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.204445 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.216142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.213213 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.216142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.213472 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.216142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.213708 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.221124 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.219821 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.221457 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.221436 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.243768 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.243641 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8b5c\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.244467 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.244434 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.304870 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.304939 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.304969 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9zl\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305027 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305129 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305164 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305219 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305325 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305376 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.305831 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.308138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.306069 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.309598 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.309572 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.310639 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.310614 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.326282 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.321408 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.326282 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.321624 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.331663 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.329054 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.389842 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.389769 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9zl\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.413152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.393692 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.433366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.433336 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:51.469649 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.469616 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:51.620925 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.619296 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2"] Oct 30 05:49:51.630724 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:49:51.630692 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f878bc_d34c_450b_a038_1956f16d58ad.slice/crio-293739758e433aca593faab6c8c595f20401b4ed2cfe7077233de6d57deb4e90 WatchSource:0}: Error finding container 293739758e433aca593faab6c8c595f20401b4ed2cfe7077233de6d57deb4e90: Status 404 returned error can't find the container with id 293739758e433aca593faab6c8c595f20401b4ed2cfe7077233de6d57deb4e90 Oct 30 05:49:51.633996 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.633957 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.634083 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.634042 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.634152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.634121 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.816237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.816135 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w"] Oct 30 05:49:51.827504 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.827479 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn"] Oct 30 05:49:51.828555 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.828511 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.828732 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.828610 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.828732 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.828657 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.833972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.833508 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.833972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.833577 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:51.833972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:51.833619 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.166827 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.165739 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" event={"ID":"d4f878bc-d34c-450b-a038-1956f16d58ad","Type":"ContainerStarted","Data":"5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca"} Oct 30 05:49:52.166827 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.165800 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" event={"ID":"d4f878bc-d34c-450b-a038-1956f16d58ad","Type":"ContainerStarted","Data":"293739758e433aca593faab6c8c595f20401b4ed2cfe7077233de6d57deb4e90"} Oct 30 05:49:52.169947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.169147 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" event={"ID":"9196a232-a912-4924-9b88-05f14a101fad","Type":"ContainerStarted","Data":"54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8"} Oct 30 05:49:52.169947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.169206 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" event={"ID":"9196a232-a912-4924-9b88-05f14a101fad","Type":"ContainerStarted","Data":"72954fe3468d7343331a5cc4a8e36704aa550df4e8938813332095b86aca077f"} Oct 30 05:49:52.180079 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.179319 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" event={"ID":"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c","Type":"ContainerStarted","Data":"7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939"} Oct 30 05:49:52.180079 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.179376 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" event={"ID":"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c","Type":"ContainerStarted","Data":"dcd995a750dac19cddef3dfb430256cd94bcf694e32fbd1a2acaf371ca916479"} Oct 30 05:49:52.219476 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.219404 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podStartSLOduration=2.219384207 podStartE2EDuration="2.219384207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:52.216338484 +0000 UTC m=+1046.324377515" watchObservedRunningTime="2026-10-30 05:49:52.219384207 +0000 UTC m=+1046.327423241" Oct 30 05:49:52.266949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.266645 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podStartSLOduration=2.266624861 podStartE2EDuration="2.266624861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:52.26376662 +0000 UTC m=+1046.371805676" watchObservedRunningTime="2026-10-30 05:49:52.266624861 +0000 UTC m=+1046.374663912" Oct 30 05:49:52.316496 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.316439 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podStartSLOduration=1.316425985 podStartE2EDuration="1.316425985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:52.314515577 +0000 UTC m=+1046.422554608" watchObservedRunningTime="2026-10-30 05:49:52.316425985 +0000 UTC m=+1046.424465017" Oct 30 05:49:52.435014 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.434922 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:52.436521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.436493 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:52.436633 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.436552 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:52.470455 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.470429 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:49:52.506171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.502297 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:52.506171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:52.502395 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:53.061130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.060696 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:53.066274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.066243 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:49:53.066565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.066534 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 05:49:53.444133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.436532 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:53.444133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.436592 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:53.485135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.481797 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:53.485135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:53.481867 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:54.050133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:54.045635 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:49:54.050133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:54.045708 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 05:49:54.435923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:54.435478 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:54.435923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:54.435540 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:54.473603 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:54.473466 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:54.473603 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:54.473535 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:55.032132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:55.029401 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:49:55.032132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:55.029461 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 05:49:55.434602 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:55.434567 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:55.434875 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:55.434839 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:55.472576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:55.472544 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:55.473201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:55.473159 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:56.039280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:56.039243 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:49:56.039618 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:56.039574 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 05:49:56.453590 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:56.453558 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:56.453908 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:56.453866 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:56.481042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:56.479729 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:56.481042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:56.479782 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:57.030794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:57.030759 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:49:57.031265 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:57.031213 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 05:49:57.434821 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:57.434787 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:57.435121 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:57.435074 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:57.484537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:57.484505 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:57.485131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:57.485077 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:58.034628 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:58.034204 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 05:49:58.034628 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:58.034273 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 05:49:58.434870 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:58.434837 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 05:49:58.435058 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:58.434899 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 05:49:58.475719 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:58.475685 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:58.476019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:58.475990 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:59.216611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.214015 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:59.216611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.215737 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:59.217418 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.217395 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 05:49:59.470219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.470144 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 05:49:59.470219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.470196 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 05:49:59.574387 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.574357 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:59.576160 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.576136 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:49:59.577554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:49:59.577534 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 05:50:00.475820 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:00.475787 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:50:00.476424 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:00.476304 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:50:00.477444 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:00.477408 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 05:50:12.913134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:12.913078 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk"] Oct 30 05:50:13.163368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.150405 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.214181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.196289 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 05:50:13.214181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.199230 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 05:50:13.243201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.243158 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.243569 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.243537 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.243710 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.243695 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.243917 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.243903 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.277175 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.247177 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.289129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.285271 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.384419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.381660 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.384419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.287349 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7ckgr\"" Oct 30 05:50:13.384419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.286237 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 05:50:13.384419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.287394 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk"] Oct 30 05:50:13.503192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.484431 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.503448 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.503422 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.503661 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.503641 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.503882 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.503865 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.503996 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.503980 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.537214 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.504256 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.579223 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.579188 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.579536 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.518526 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.579647 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.534687 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.579728 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.557129 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.580217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.506725 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.580376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.563622 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.580464 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.568545 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.580545 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.568872 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.580908 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.563212 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.646072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.628968 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.667850 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.667764 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xfmtk\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.709161 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.684049 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.709161 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.684273 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:13.751336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.751254 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.757510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.756597 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.993533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.942737 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.993533 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:50:13.988477 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:50:13.993533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:13.989460 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:50:14.113950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.112923 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:50:14.604171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.589672 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.627191 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.696841 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.698850 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.777206 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.777528 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.823913 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.825026 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.830388 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:50:14.848206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.831910 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:50:14.857921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.857888 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:50:14.862610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.861791 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:50:14.976252 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.976206 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.156510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:14.993915 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.157990 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.132615 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:50:15.197131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.168203 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk"] Oct 30 05:50:15.197131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.168315 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:50:15.197131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.168358 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:50:15.197131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.168394 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:50:15.285671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.258601 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:50:15.432308 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.432223 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:15.432539 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.432510 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.703198 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.702934 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.733356 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.704812 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.733356 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.705167 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:50:15.785678 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.785653 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:50:15.815245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.815201 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.841439 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.837181 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.869126 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.861282 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.888194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:15.828195 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.127366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.122732 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:50:16.127366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.122820 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:50:16.250801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.246594 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:50:16.250801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.246636 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:50:16.250801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.246673 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:50:16.250801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.246691 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:50:16.250801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.246712 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:50:16.250801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:16.247171 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" event={"ID":"49638c53-13ed-4a7e-8dc5-c5942c9117fe","Type":"ContainerStarted","Data":"1a7650f31b70e9186dd57628181c62acd1817edea52d29afa87fd4eb83017be4"} Oct 30 05:50:18.529741 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:18.529704 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:18.530044 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:18.529770 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:19.285833 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:19.285793 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" event={"ID":"49638c53-13ed-4a7e-8dc5-c5942c9117fe","Type":"ContainerStarted","Data":"890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788"} Oct 30 05:50:19.286528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:19.286474 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:19.288247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:19.288223 5980 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xfmtk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:50:19.288486 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:19.288464 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" podUID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:50:19.350523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:19.350470 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" podStartSLOduration=4.336130877 podStartE2EDuration="7.350453784s" totalImagesPullingTime="3.014322907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:19.347914117 +0000 UTC m=+1073.455953150" watchObservedRunningTime="2026-10-30 05:50:19.350453784 +0000 UTC m=+1073.458492815" Oct 30 05:50:20.294426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:20.294397 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 05:50:21.093612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.093582 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh"] Oct 30 05:50:21.098026 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.097995 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.127069 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.127042 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh"] Oct 30 05:50:21.237214 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237184 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237264 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237333 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237362 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237446 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwn8\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237541 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237519 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237594 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237576 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237680 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.237730 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.237710 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.338781 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.338752 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwn8\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.338781 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.338786 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.338821 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.338922 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.338961 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339081 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339162 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339237 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339270 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339302 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339466 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339529 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.339940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339548 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.340098 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.339947 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.341329 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.341313 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.341570 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.341554 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.379409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.379351 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.379409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.379385 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwn8\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.416504 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.416485 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:21.743576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.743549 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh"] Oct 30 05:50:21.745197 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:50:21.745172 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb793ae0f_d086_4c38_af08_d68376033083.slice/crio-31e709103e413d005a8b363f7846a08b90183fd4c94f6cd699d7b33ce5ea9c54 WatchSource:0}: Error finding container 31e709103e413d005a8b363f7846a08b90183fd4c94f6cd699d7b33ce5ea9c54: Status 404 returned error can't find the container with id 31e709103e413d005a8b363f7846a08b90183fd4c94f6cd699d7b33ce5ea9c54 Oct 30 05:50:21.751361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.750982 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:21.751361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.751148 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:21.751361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:21.751237 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:22.308291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:22.308195 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" event={"ID":"b793ae0f-d086-4c38-af08-d68376033083","Type":"ContainerStarted","Data":"2cff76b764d09b394473cac8665d275dabecbe579539474fc133525da4c54fcb"} Oct 30 05:50:22.308291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:22.308258 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" event={"ID":"b793ae0f-d086-4c38-af08-d68376033083","Type":"ContainerStarted","Data":"31e709103e413d005a8b363f7846a08b90183fd4c94f6cd699d7b33ce5ea9c54"} Oct 30 05:50:22.417631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:22.417602 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:22.419604 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:22.419578 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:50:22.419728 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:22.419628 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 05:50:23.417907 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:23.417865 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:50:23.418350 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:23.417938 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 05:50:24.417873 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:24.417842 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:50:24.418044 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:24.417889 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 05:50:25.417773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:25.417737 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:50:25.417984 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:25.417792 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 05:50:26.416837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:26.416806 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:50:26.417230 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:26.416859 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 05:50:27.417732 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:27.417700 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:50:27.418170 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:27.417763 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 05:50:28.556823 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:28.556788 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:28.557968 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:28.557940 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:28.558798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:28.558777 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 05:50:28.619264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:28.619222 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podStartSLOduration=7.619207713 podStartE2EDuration="7.619207713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:22.388561339 +0000 UTC m=+1076.496600372" watchObservedRunningTime="2026-10-30 05:50:28.619207713 +0000 UTC m=+1082.727246745" Oct 30 05:50:35.735632 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.735542 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f"] Oct 30 05:50:35.743478 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.743450 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.783061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.783005 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f"] Oct 30 05:50:35.870479 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870433 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.870787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870558 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspf6\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.870787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870586 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.870787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870673 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.870787 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870704 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.871014 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870812 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.871014 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870882 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.871014 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870938 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.871014 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.870976 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.972852 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.972817 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.972873 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.972935 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.972987 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973222 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973146 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973222 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973185 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973254 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973395 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973345 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973447 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973404 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973505 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973467 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspf6\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973558 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973503 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973558 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973543 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973794 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.973957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.973807 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.975486 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.975466 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.975855 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.975838 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.997666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.997607 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:35.997666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:35.997618 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspf6\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:36.058965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.058940 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:36.386357 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.386330 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f"] Oct 30 05:50:36.387738 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:50:36.387709 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ab37b5_d172_4357_82c6_1e17249a4d7f.slice/crio-396473051e7cc7086ef4d34a449b3f3c669bffb1588ae620092821d149c30401 WatchSource:0}: Error finding container 396473051e7cc7086ef4d34a449b3f3c669bffb1588ae620092821d149c30401: Status 404 returned error can't find the container with id 396473051e7cc7086ef4d34a449b3f3c669bffb1588ae620092821d149c30401 Oct 30 05:50:36.390424 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.390388 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.390514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.390453 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.390514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.390482 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.410150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.409901 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" event={"ID":"35ab37b5-d172-4357-82c6-1e17249a4d7f","Type":"ContainerStarted","Data":"396473051e7cc7086ef4d34a449b3f3c669bffb1588ae620092821d149c30401"} Oct 30 05:50:36.516877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.515571 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2"] Oct 30 05:50:36.522506 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.519983 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.559217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.559074 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2"] Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581570 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581628 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581788 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581814 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581841 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581866 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581897 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.581958 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfkfm\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.582396 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.582014 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.683693 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.683606 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.683859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.683750 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.683859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.683794 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.683984 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.683956 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.683981 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684008 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684031 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684065 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684140 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfkfm\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684199 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684439 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684652 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684525 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.684704 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.684685 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.685073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.685045 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.686803 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.686768 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.687555 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.687529 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.712141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.712095 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.713297 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.713275 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfkfm\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:36.849863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:36.849835 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:37.185762 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.185722 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2"] Oct 30 05:50:37.212485 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.208947 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:37.212485 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.209032 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:37.212485 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.209073 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:37.426510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.426467 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" event={"ID":"35ab37b5-d172-4357-82c6-1e17249a4d7f","Type":"ContainerStarted","Data":"975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045"} Oct 30 05:50:37.429085 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.429055 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" event={"ID":"3c3b2678-6a74-406f-89c9-e3247e5fc2b9","Type":"ContainerStarted","Data":"e44e15d66a0e302e0b215c2d4b52194d3c68565f099671d13a0fe09b599bde6d"} Oct 30 05:50:37.429245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.429121 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" event={"ID":"3c3b2678-6a74-406f-89c9-e3247e5fc2b9","Type":"ContainerStarted","Data":"5f73ac2c80846692855344f563ff3d782513b75d25544f52ca325ade575c5527"} Oct 30 05:50:37.501525 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.501403 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podStartSLOduration=2.5013835650000003 podStartE2EDuration="2.501383565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:37.499685763 +0000 UTC m=+1091.607724795" watchObservedRunningTime="2026-10-30 05:50:37.501383565 +0000 UTC m=+1091.609422598" Oct 30 05:50:37.854130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:37.850181 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:38.069170 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:38.059980 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:38.069170 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:38.065588 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.069170 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:38.065644 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:38.852474 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:38.852443 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.852794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:38.852499 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:39.069291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:39.062473 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.069291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:39.062535 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:39.858135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:39.853493 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.858135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:39.853558 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:40.059883 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.059453 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.059883 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.059509 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:40.147061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.146965 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.198847 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.198287 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.237692 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.237637 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.708274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.708224 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.869182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.841292 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:50:40.869182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.867203 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.869182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.867631 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:50:40.920211 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.903413 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.985196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.974217 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.985196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:40.974283 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:41.111624 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.111586 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.117059 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.111840 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:41.134180 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.117376 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.166881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.166840 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:50:41.223166 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.221942 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:50:41.243406 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.243306 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.243926 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.243895 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.244347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.244330 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:50:41.245405 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.245379 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.245743 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.245726 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:50:41.393135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.389455 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podStartSLOduration=5.389435167 podStartE2EDuration="5.389435167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:37.550286515 +0000 UTC m=+1091.658325564" watchObservedRunningTime="2026-10-30 05:50:41.389435167 +0000 UTC m=+1095.497474199" Oct 30 05:50:41.713323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.713273 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.782002 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.764493 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:50:41.896195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.870169 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.896195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.870225 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:41.896195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.870608 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.913133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.897058 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.948483 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.919212 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.948861 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.948841 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:50:41.984265 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.984175 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:50:41.997135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.993820 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:50:42.078194 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:41.920510 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.105585 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.105553 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:42.105855 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.105827 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:42.179778 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.179747 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:50:42.244192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.208157 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.255201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.255148 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.279141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.255540 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.304585 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.258328 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.344288 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.344260 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:50:42.720262 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.720216 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.855274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.855227 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:42.855570 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.855534 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:42.890660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.890292 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:50:42.950647 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.950261 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:50:42.997351 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:42.996887 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.116214 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.116156 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.208231 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.135463 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:43.208779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.208739 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:43.208898 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.177690 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:43.208999 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.208982 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.259286 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.259187 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.264131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.259487 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:50:43.362699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.362656 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:50:43.381133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.364075 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.694130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.652423 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 05:50:43.811158 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.738747 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 05:50:43.811158 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.767316 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.851823 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.851779 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:43.877294 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.852072 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:43.891316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.891258 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.958342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:43.958230 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.034290 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.015268 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.085827 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.085788 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:44.086087 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.086061 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:44.200575 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:50:44.200313 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:50:44.209185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.208197 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:50:44.290353 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.290291 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:50:44.291989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.291941 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.369271 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.369224 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.492196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.483894 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.492196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.483955 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.510240 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.510204 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.510496 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.510472 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.693134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.689647 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:50:44.714289 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.714256 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.714597 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.714553 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.730837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.730789 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.731142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.731081 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.772824 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.772696 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.851375 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.851342 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:44.852008 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.851605 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:44.888273 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.888232 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.888560 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.888529 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.937685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.937595 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.959710 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:44.959660 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.060205 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.060086 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:45.060468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.060425 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:45.072061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.072016 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.323466 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.323418 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:50:45.325244 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.325176 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:50:45.326199 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.326170 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:50:45.381411 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.381354 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.711937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.711222 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:50:45.779374 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.778209 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:50:45.795381 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.795250 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:45.795381 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.795306 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.854076 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.854045 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:45.856260 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.855844 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:45.941272 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.941199 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.960674 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:45.960630 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.066281 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.065317 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:46.066281 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.065378 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:46.081318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.081212 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.210537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.210207 5980 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:46.210537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.210273 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.304891 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.304767 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:50:46.339215 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.336173 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:50:46.340281 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.340251 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:50:46.360435 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.340360 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.385524 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.382476 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.390423 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.390396 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:50:46.555134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.546965 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.623789 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.623552 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:50:46.712231 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.712187 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:50:46.719572 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.719546 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:50:46.859406 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.859365 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:46.859669 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.859644 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:47.034198 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.982969 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:50:47.034198 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:46.987371 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:50:47.034198 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.020937 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:50:47.092515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.060393 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:47.092515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.060446 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:47.092515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.060650 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:50:47.092515 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.062385 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:50:47.151054 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.151026 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:50:47.154217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.153943 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:50:47.850713 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.850682 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:47.851123 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:47.850731 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:48.059776 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:48.059742 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 05:50:48.059929 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:48.059790 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 05:50:48.850779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:48.850746 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:50:48.851245 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:48.850809 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 05:50:49.258894 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:49.258428 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:49.260147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:49.260123 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:49.260526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:49.260511 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:50:49.855102 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:49.855072 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:49.855568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:49.855531 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:50:49.856356 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:50:49.856340 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:51:06.311130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.295999 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x"] Oct 30 05:51:06.365742 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.365704 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x"] Oct 30 05:51:06.382130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.366154 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.471675 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.471637 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.472009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.471986 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.472189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.472169 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.472323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.472305 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.472560 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.472540 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphbz\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.472707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.472692 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.472859 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.472842 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.473046 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.473029 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.473224 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.473207 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.596142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.586061 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.601368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.601328 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.602124 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.601959 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.603022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.602970 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.628132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.622172 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.628132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.622570 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphbz\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.628132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.623268 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.628132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.623378 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.628132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.623545 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.628132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.623623 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.638343 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.629035 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.649708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.636586 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.649708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.636738 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.649956 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.639218 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.785311 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.785227 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.855621 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.834938 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphbz\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.878330 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.878262 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:06.890570 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:06.890538 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:07.111947 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.111642 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:07.265662 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.265604 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.324649 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.319754 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:07.436812 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.436705 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:51:07.695714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.695608 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/health\": context deadline exceeded" Oct 30 05:51:07.938135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.936201 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.003191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:07.998272 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.023146 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.009086 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:51:08.164131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.156214 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.170130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.166294 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.293885 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.289204 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.293885 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.289743 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:51:08.293885 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.290223 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:51:08.354145 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.337253 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:51:08.354145 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.337770 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:08.354145 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.338205 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:51:08.584196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.537066 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.611134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.597594 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:51:08.790133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.779675 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:51:09.004190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:08.989090 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.178133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.084652 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:51:09.290331 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.290242 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:51:09.706439 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.336609 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.716900 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.483726 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:51:09.744228 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.483780 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:09.746062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.746040 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:51:09.772043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.556222 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:51:09.775220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.590977 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:51:09.775220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.592382 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.775220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.772876 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:51:09.775220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.772960 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:51:09.775220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.594069 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:51:09.807149 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.775944 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:51:09.807149 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.789301 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:51:09.807149 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:09.789705 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:51:10.089197 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.058975 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 05:51:10.215496 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.156616 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:10.215777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.182640 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:51:10.632141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.615220 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:10.713816 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.713763 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:10.735048 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.735004 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.058773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.849310 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.060673 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.850624 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:51:11.095471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.850663 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:11.150134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.850684 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:51:11.150134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.850705 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:51:11.150134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.850722 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:51:11.150134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:10.850739 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:51:11.250209 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:51:11.250093 5980 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.093s" Oct 30 05:51:11.250436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:11.250419 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x"] Oct 30 05:51:11.266784 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:11.266747 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" event={"ID":"61eb1e21-74fd-4bdf-9782-fa4a081782e3","Type":"ContainerStarted","Data":"71ff45bf94488b1536745995364ddc051621be2dc7d4f380355c0bde31377929"} Oct 30 05:51:11.432265 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:11.432218 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 05:51:11.670946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:11.670896 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.938837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:11.938751 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:51:12.019130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:11.988453 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.073264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.073212 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.073896 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.073862 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.117260 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.117214 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.207267 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.120192 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:12.207938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.120252 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.208936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.120277 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.208936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.120299 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:51:12.757900 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.757873 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:51:12.985191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.951236 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:51:12.985191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.961335 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:51:12.985191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:12.970925 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:51:13.076692 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.074986 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:13.076692 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.075262 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:51:13.111146 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.102149 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:51:13.131126 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.115019 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:51:13.159443 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.157171 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:51:13.164152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.161521 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:51:13.166139 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.166016 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:51:13.201313 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:51:13.201280 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:51:13.654375 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.631295 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" event={"ID":"61eb1e21-74fd-4bdf-9782-fa4a081782e3","Type":"ContainerStarted","Data":"3d9531426d233f95eebf51ce413a51bac8832676b99d67568f4ce4d66ebaaed9"} Oct 30 05:51:13.728142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:13.724961 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podStartSLOduration=7.724935938 podStartE2EDuration="7.724935938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:13.682321767 +0000 UTC m=+1127.790360798" watchObservedRunningTime="2026-10-30 05:51:13.724935938 +0000 UTC m=+1127.832974970" Oct 30 05:51:14.117132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:14.116599 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:14.119213 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:14.119186 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 05:51:14.119453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:14.119427 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 05:51:15.112830 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:15.112795 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 05:51:15.113081 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:15.113056 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 05:51:16.113205 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:16.113174 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 05:51:16.113590 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:16.113225 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 05:51:17.112994 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:17.112965 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 05:51:17.113201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:17.113021 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 05:51:18.113099 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:18.113069 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 05:51:18.113491 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:18.113140 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 05:51:19.113432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:19.113397 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 05:51:19.113832 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:19.113451 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 05:51:20.507252 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:20.506947 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:20.508159 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:20.508132 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:20.509149 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:20.509129 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 05:51:27.759135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:27.757523 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.967164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:27.945244 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.998527 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:27.998481 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.097128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.074347 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 05:51:28.097128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.074858 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.097128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.079178 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.148147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.147423 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:51:28.163978 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.163931 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.234132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.221734 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.769255 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.769205 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:51:28.770937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.770905 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:28.787275 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.787249 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:51:28.886255 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.833785 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.935262 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.914071 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.947410 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.947370 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.947950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.947928 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:51:29.011576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:28.999912 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.017439 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.012593 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:51:29.237133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.230754 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.237133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.231124 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.271137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.239158 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:51:29.338371 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.338341 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:51:29.345315 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.262859 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:51:29.345315 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.262522 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:51:29.394344 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.366277 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:51:29.394344 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.366384 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:51:29.485195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.466138 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:29.485195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.466194 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.788671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.788362 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.835459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.833233 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.835459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.833553 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:51:29.835459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.834860 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.835459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.835069 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:51:29.861301 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:29.847247 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.023277 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.023232 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:51:30.032246 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.032207 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.230337 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.230309 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:51:30.230788 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.230773 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:51:30.230954 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.230928 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.236493 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.233045 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.237771 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.237656 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:51:30.262955 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.262930 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:51:30.310661 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.310638 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:51:30.338376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.338354 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:51:30.365359 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.365321 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:51:30.378117 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.378070 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:51:30.404691 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:30.404660 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:51:36.063288 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.063258 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552"] Oct 30 05:51:36.068914 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.068890 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.099815 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.099777 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552"] Oct 30 05:51:36.210419 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210381 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210432 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210526 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210620 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210660 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210714 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210886 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210789 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210886 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210863 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.210981 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.210926 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs84k\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312094 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312365 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312181 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs84k\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312442 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312385 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312501 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312446 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312563 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312484 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312563 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312554 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312567 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312595 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312626 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.312831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312708 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.313010 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.312981 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.313127 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.313070 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.313486 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.313265 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.313486 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.313294 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.314978 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.314955 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.315152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.315131 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.335559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.335527 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.335740 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.335719 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs84k\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.398421 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.398387 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:36.737076 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.737048 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552"] Oct 30 05:51:36.738188 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:51:36.738164 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode220f18d_16fd_406b_85d4_0f1250206231.slice/crio-00e3c2834b26808522fd58856320d0d4f3843000e7bfe4057a5335701d2ef481 WatchSource:0}: Error finding container 00e3c2834b26808522fd58856320d0d4f3843000e7bfe4057a5335701d2ef481: Status 404 returned error can't find the container with id 00e3c2834b26808522fd58856320d0d4f3843000e7bfe4057a5335701d2ef481 Oct 30 05:51:36.740571 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.740528 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.740649 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.740596 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.740649 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.740626 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.753631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.753598 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2"] Oct 30 05:51:36.760404 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.760348 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.783852 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.783819 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2"] Oct 30 05:51:36.820290 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820244 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.820428 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820300 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.820802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820474 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.820802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820557 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.820802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820677 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.820802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820779 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.821261 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820866 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.821261 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.820935 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtdqm\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.821261 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.821035 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922614 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922575 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922824 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922629 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922824 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922680 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922824 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922777 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922992 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922823 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922992 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922893 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtdqm\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.922992 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922928 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.923162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.922991 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.923162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.923023 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.923162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.923040 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.923837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.923412 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.923837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.923569 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.923837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.923763 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.924027 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.923921 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.925378 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.925356 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.925963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.925942 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.946703 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.946678 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:36.946884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:36.946864 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtdqm\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:37.092651 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.092617 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:37.425042 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:51:37.425008 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9e93d3_9828_4fce_ad65_9a3e81ce9c8d.slice/crio-6c02bc6bcf8c8227f7d6191a76dab4447bbd4d0cf99324c67bc4f8b1863f86ff WatchSource:0}: Error finding container 6c02bc6bcf8c8227f7d6191a76dab4447bbd4d0cf99324c67bc4f8b1863f86ff: Status 404 returned error can't find the container with id 6c02bc6bcf8c8227f7d6191a76dab4447bbd4d0cf99324c67bc4f8b1863f86ff Oct 30 05:51:37.427292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.427264 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2"] Oct 30 05:51:37.427878 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.427847 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:37.427921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.427909 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:37.427957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.427938 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:37.584132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.566326 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" event={"ID":"e220f18d-16fd-406b-85d4-0f1250206231","Type":"ContainerStarted","Data":"040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe"} Oct 30 05:51:37.584132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.566389 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" event={"ID":"e220f18d-16fd-406b-85d4-0f1250206231","Type":"ContainerStarted","Data":"00e3c2834b26808522fd58856320d0d4f3843000e7bfe4057a5335701d2ef481"} Oct 30 05:51:37.607132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.585084 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" event={"ID":"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d","Type":"ContainerStarted","Data":"8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4"} Oct 30 05:51:37.607132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.585162 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" event={"ID":"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d","Type":"ContainerStarted","Data":"6c02bc6bcf8c8227f7d6191a76dab4447bbd4d0cf99324c67bc4f8b1863f86ff"} Oct 30 05:51:37.624585 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.624501 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podStartSLOduration=1.624480826 podStartE2EDuration="1.624480826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:37.621788793 +0000 UTC m=+1151.729827863" watchObservedRunningTime="2026-10-30 05:51:37.624480826 +0000 UTC m=+1151.732519859" Oct 30 05:51:37.728214 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:37.719983 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podStartSLOduration=1.719965185 podStartE2EDuration="1.719965185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:37.71858972 +0000 UTC m=+1151.826628782" watchObservedRunningTime="2026-10-30 05:51:37.719965185 +0000 UTC m=+1151.828004218" Oct 30 05:51:38.095128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:38.094255 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:38.099130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:38.096071 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:38.099130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:38.096186 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:38.403355 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:38.403257 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:38.413303 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:38.413268 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:51:38.413568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:38.413541 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:51:39.100133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.098560 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:39.100133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.098627 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:39.410610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.410520 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:51:39.410868 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.410842 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:51:39.925641 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.925602 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn"] Oct 30 05:51:39.930030 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.930004 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:39.939354 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.939332 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn"] Oct 30 05:51:39.951962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.951935 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-709c4831\"" Oct 30 05:51:39.951962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:39.951956 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-709c4831-dockercfg-jvchg\"" Oct 30 05:51:40.041864 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.041830 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjvt\" (UniqueName: \"kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt\") pod \"payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.042069 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.041951 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.043785 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.043756 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd"] Oct 30 05:51:40.048773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.048751 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd"] Oct 30 05:51:40.048944 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.048933 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.130393 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.130365 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:40.145609 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.145572 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:40.146249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.146069 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume\") pod \"payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.146470 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.146436 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjvt\" (UniqueName: \"kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt\") pod \"payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.147023 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.146957 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.149371 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.149332 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctq7\" (UniqueName: \"kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7\") pod \"payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.150022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.149077 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.349361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.349324 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume\") pod \"payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.349872 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.349850 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctq7\" (UniqueName: \"kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7\") pod \"payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.351238 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.351204 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume\") pod \"payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.425025 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.424991 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:51:40.425326 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.425293 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:51:40.426042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.426006 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjvt\" (UniqueName: \"kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt\") pod \"payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.500191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.481318 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctq7\" (UniqueName: \"kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7\") pod \"payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:40.557160 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.550385 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:40.740991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:40.740944 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:41.130136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.128283 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:41.130136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.128374 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:41.359156 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.358921 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn"] Oct 30 05:51:41.403141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.399472 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:51:41.403141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.399543 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:51:41.425680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.425645 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd"] Oct 30 05:51:41.713087 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.713043 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" event={"ID":"d2162fae-53cf-45e5-84aa-cd08baefde3f","Type":"ContainerStarted","Data":"16eb2f3dc801ffee82bb4f2efb36c5f2eb0bef1b570a220e82fce8e304eb119f"} Oct 30 05:51:41.715320 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.715268 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" event={"ID":"d2162fae-53cf-45e5-84aa-cd08baefde3f","Type":"ContainerStarted","Data":"3271f09963a69db67e8756760fbe14e6ecf002d6dc8692236f4c432bb61c512d"} Oct 30 05:51:41.715794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.715757 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:41.715949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.715933 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" event={"ID":"38a6af8b-ac86-4edb-be94-fc9a35ebaa39","Type":"ContainerStarted","Data":"86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421"} Oct 30 05:51:41.716063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.716050 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" event={"ID":"38a6af8b-ac86-4edb-be94-fc9a35ebaa39","Type":"ContainerStarted","Data":"8af6518fe9907eb383b9ad97950452fcd0c4b14e9b238b72eba83ca16862ab79"} Oct 30 05:51:41.723312 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.716748 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:41.759608 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:41.759557 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" podStartSLOduration=2.759544365 podStartE2EDuration="2.759544365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:41.757576191 +0000 UTC m=+1155.865615223" watchObservedRunningTime="2026-10-30 05:51:41.759544365 +0000 UTC m=+1155.867583396" Oct 30 05:51:42.099128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:42.096463 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:42.099128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:42.096522 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:42.402040 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:42.401498 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:51:42.402040 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:42.401576 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:51:43.093533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:43.093502 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:43.093759 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:43.093563 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:43.122711 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:51:43.122671 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:51:43.401217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:43.399483 5980 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:51:43.401217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:43.399554 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:51:44.099151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:44.096396 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:44.099151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:44.096480 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:44.711100 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:44.711067 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:44.712138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:44.712077 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:44.713364 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:44.713339 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 05:51:44.783653 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:44.783601 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" podStartSLOduration=5.783587084 podStartE2EDuration="5.783587084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:41.814075784 +0000 UTC m=+1155.922114821" watchObservedRunningTime="2026-10-30 05:51:44.783587084 +0000 UTC m=+1158.891626116" Oct 30 05:51:45.093912 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:45.093875 5980 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 05:51:45.094092 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:45.093937 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 05:51:46.098964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:46.098923 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:46.099773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:46.099751 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:46.102650 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:46.102628 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 05:51:50.973388 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:50.973355 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr"] Oct 30 05:51:50.977455 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:50.977436 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.000335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.000298 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr"] Oct 30 05:51:51.011367 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.002850 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-54d4943b-serving-cert\"" Oct 30 05:51:51.056512 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.056484 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.056650 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.056538 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.056799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.056777 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.056837 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.056826 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdnt\" (UniqueName: \"kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.158031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.157998 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.158336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.158307 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdnt\" (UniqueName: \"kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.158462 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.158444 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.158517 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.158502 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.158760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.158726 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.161255 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.161232 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.161734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.161714 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.184807 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.183709 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdnt\" (UniqueName: \"kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt\") pod \"maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.304047 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.303950 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:51.635656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.635626 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr"] Oct 30 05:51:51.635817 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:51:51.635667 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4073f0c_82dc_4527_a896_20104fe5508d.slice/crio-a7bcbd1137834a14e543eff3a1c0fd15c07b91a2075a3f729e0f4309110106a0 WatchSource:0}: Error finding container a7bcbd1137834a14e543eff3a1c0fd15c07b91a2075a3f729e0f4309110106a0: Status 404 returned error can't find the container with id a7bcbd1137834a14e543eff3a1c0fd15c07b91a2075a3f729e0f4309110106a0 Oct 30 05:51:51.793510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:51.793469 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" event={"ID":"d4073f0c-82dc-4527-a896-20104fe5508d","Type":"ContainerStarted","Data":"a7bcbd1137834a14e543eff3a1c0fd15c07b91a2075a3f729e0f4309110106a0"} Oct 30 05:51:52.803456 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:52.803416 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" event={"ID":"d4073f0c-82dc-4527-a896-20104fe5508d","Type":"ContainerStarted","Data":"8610f5bd6585958caddbc27b64f44fa08910955f200f874eeedb812575cec14f"} Oct 30 05:51:52.803936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:52.803486 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:52.855911 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:52.855309 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" podStartSLOduration=2.4619136839999998 podStartE2EDuration="2.855287918s" totalImagesPullingTime="393.374234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:52.848997338 +0000 UTC m=+1166.957036372" watchObservedRunningTime="2026-10-30 05:51:52.855287918 +0000 UTC m=+1166.963326970" Oct 30 05:51:53.738782 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:53.738752 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:51:53.738946 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:53.738821 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:51:56.265009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:56.264962 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:51:56.293248 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:56.293204 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:56.412416 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:56.410222 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:51:56.545701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:56.545234 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded" Oct 30 05:51:57.068907 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.067436 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.234130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.233316 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.291598 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.291530 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.354627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.343578 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.354627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.292690 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.354627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.344193 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:51:57.354627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.344266 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:51:57.470763 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.470719 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.707264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:57.471154 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:51:58.096845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.093208 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.096845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.093524 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.270141 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.270086 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:51:58.288397 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.274952 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 05:51:58.292208 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.292176 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.350185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.344098 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.401129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.375988 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:51:58.471190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.471169 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:51:58.474383 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.474361 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:51:58.482378 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.471039 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.562576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.384295 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.839129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:58.835962 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.103188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.100194 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.151531 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.151498 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:51:59.257404 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.257328 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.290378 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.290339 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.324336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.324292 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.366237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.345613 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.444541 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.372726 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.622189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.597670 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.686661 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.686526 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.868589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:51:59.868535 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.192133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.140921 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.260133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.258559 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.312136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.308048 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.374132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.374067 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.425974 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.401716 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:52:00.456179 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.401751 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 05:52:00.512134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.509859 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:52:00.540255 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.540213 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 05:52:00.540941 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.540916 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:52:00.630239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.630197 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.771193 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.699773 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.971132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.944977 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:52:00.971132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.958440 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 05:52:00.971132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.959869 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 05:52:00.981011 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:00.980988 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 05:52:01.060081 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.059998 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 05:52:01.334878 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.334850 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:52:01.427137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.426787 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.432139 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.429734 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:52:01.458267 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.458234 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.620873 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.620794 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 05:52:01.648798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.648743 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.676972 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.676944 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:52:01.939702 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:01.937889 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:52:09.670366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.670330 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8"] Oct 30 05:52:09.676207 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.676187 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.698492 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.698465 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8"] Oct 30 05:52:09.705695 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.705675 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6b5b1e-openshift-default-dockercfg-n2kmp\"" Oct 30 05:52:09.771994 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.771962 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772259 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772138 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772259 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772226 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772410 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772383 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772686 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772661 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwg9n\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772806 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772734 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772806 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772779 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772932 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772864 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.772985 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.772929 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875222 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwg9n\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875282 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875311 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875382 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875455 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875525 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875612 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875683 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876138 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.875737 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876715 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.876254 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.876854 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.876828 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.901129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.895700 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.901129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.896356 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.901129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.896939 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.934284 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.933340 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:09.994000 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:09.993967 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:10.003721 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.003688 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwg9n\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:10.013153 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.007397 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:10.302328 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.296019 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:10.660088 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.660063 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8"] Oct 30 05:52:10.663552 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:52:10.663516 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda274fed5_ff91_44aa_a19f_129c6b9c8b32.slice/crio-87bc74a51464a9184ccbc8f352836fc8675648a5850395f877ec5e2d670b119d WatchSource:0}: Error finding container 87bc74a51464a9184ccbc8f352836fc8675648a5850395f877ec5e2d670b119d: Status 404 returned error can't find the container with id 87bc74a51464a9184ccbc8f352836fc8675648a5850395f877ec5e2d670b119d Oct 30 05:52:10.667399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.667357 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:52:10.667512 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.667468 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:52:10.667568 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:10.667514 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:52:11.300025 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:11.299982 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" event={"ID":"a274fed5-ff91-44aa-a19f-129c6b9c8b32","Type":"ContainerStarted","Data":"6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2"} Oct 30 05:52:11.300574 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:11.300041 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" event={"ID":"a274fed5-ff91-44aa-a19f-129c6b9c8b32","Type":"ContainerStarted","Data":"87bc74a51464a9184ccbc8f352836fc8675648a5850395f877ec5e2d670b119d"} Oct 30 05:52:11.363418 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:11.363367 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podStartSLOduration=2.36335299 podStartE2EDuration="2.36335299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:11.361505971 +0000 UTC m=+1185.469545003" watchObservedRunningTime="2026-10-30 05:52:11.36335299 +0000 UTC m=+1185.471392021" Oct 30 05:52:12.297012 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:12.296976 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:12.303998 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:12.303975 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:12.309574 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:12.309541 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:12.310291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:12.310266 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:52:13.089135 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:52:13.089071 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:52:27.977232 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:27.977183 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:52:28.062994 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.062951 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.063656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.063267 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.170061 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.170020 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.326131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.323148 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 05:52:28.348130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.337453 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.552142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.550856 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded" Oct 30 05:52:28.644186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.641547 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.644186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.641860 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 05:52:28.837133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:28.836930 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:29.329175 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.329133 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:29.352013 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.337487 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:52:29.353024 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.352989 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:29.355577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.353644 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:52:29.459386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.459039 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" Oct 30 05:52:29.551236 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.551201 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:29.552666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.552644 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:52:29.577315 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.577293 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 05:52:29.800958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.800862 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:29.953130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:29.934020 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:52:30.238494 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:30.236205 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:30.240292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:30.238745 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 05:52:30.854239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:30.854209 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 05:52:34.615910 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.615879 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd"] Oct 30 05:52:34.657425 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.657393 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn"] Oct 30 05:52:34.869441 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.869332 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8"] Oct 30 05:52:34.883538 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.883511 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:34.896801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.896767 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8"] Oct 30 05:52:34.900518 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.900486 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6b5b1e-serving-cert\"" Oct 30 05:52:34.938248 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.938220 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4lz\" (UniqueName: \"kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:34.938442 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.938394 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:34.938558 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.938538 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:34.938688 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:34.938675 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.039549 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.039513 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.039742 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.039582 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4lz\" (UniqueName: \"kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.039742 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.039693 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.039862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.039751 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.040472 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.040367 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.042272 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.042249 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.042354 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.042246 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.064389 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.064360 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4lz\" (UniqueName: \"kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz\") pod \"maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.233817 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.233734 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:35.574616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.574589 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8"] Oct 30 05:52:35.576237 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:52:35.576206 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41fd892_f798_4f67_8427_65f1601efa58.slice/crio-cf0883c559d112a10d62283bfabdacf71d697b36162c783db097dc2e8d1838e7 WatchSource:0}: Error finding container cf0883c559d112a10d62283bfabdacf71d697b36162c783db097dc2e8d1838e7: Status 404 returned error can't find the container with id cf0883c559d112a10d62283bfabdacf71d697b36162c783db097dc2e8d1838e7 Oct 30 05:52:35.676130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:35.675262 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" event={"ID":"b41fd892-f798-4f67-8427-65f1601efa58","Type":"ContainerStarted","Data":"cf0883c559d112a10d62283bfabdacf71d697b36162c783db097dc2e8d1838e7"} Oct 30 05:52:36.358149 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.358096 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr"] Oct 30 05:52:36.358676 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.358630 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" podUID="d4073f0c-82dc-4527-a896-20104fe5508d" containerName="maas-api" containerID="cri-o://8610f5bd6585958caddbc27b64f44fa08910955f200f874eeedb812575cec14f" gracePeriod=30 Oct 30 05:52:36.691819 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.691739 5980 generic.go:356] "Generic (PLEG): container finished" podID="d4073f0c-82dc-4527-a896-20104fe5508d" containerID="8610f5bd6585958caddbc27b64f44fa08910955f200f874eeedb812575cec14f" exitCode=0 Oct 30 05:52:36.692284 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.691922 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" event={"ID":"d4073f0c-82dc-4527-a896-20104fe5508d","Type":"ContainerDied","Data":"8610f5bd6585958caddbc27b64f44fa08910955f200f874eeedb812575cec14f"} Oct 30 05:52:36.709724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.709685 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" event={"ID":"b41fd892-f798-4f67-8427-65f1601efa58","Type":"ContainerStarted","Data":"6e28428d85565845f28ab7bcf32a1132b1109f974032efd1d280b393884f495e"} Oct 30 05:52:36.710927 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.710912 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:36.764696 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.764437 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" podStartSLOduration=2.376011412 podStartE2EDuration="2.764416869s" totalImagesPullingTime="388.405457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:36.760252386 +0000 UTC m=+1210.868291419" watchObservedRunningTime="2026-10-30 05:52:36.764416869 +0000 UTC m=+1210.872455901" Oct 30 05:52:36.810091 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.810066 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:52:36.885579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.884717 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls\") pod \"d4073f0c-82dc-4527-a896-20104fe5508d\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " Oct 30 05:52:36.885579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.884861 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs\") pod \"d4073f0c-82dc-4527-a896-20104fe5508d\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " Oct 30 05:52:36.885579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.884922 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt\" (UniqueName: \"kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt\") pod \"d4073f0c-82dc-4527-a896-20104fe5508d\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " Oct 30 05:52:36.885579 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.885067 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca\") pod \"d4073f0c-82dc-4527-a896-20104fe5508d\" (UID: \"d4073f0c-82dc-4527-a896-20104fe5508d\") " Oct 30 05:52:36.886468 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.886444 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca" pod "d4073f0c-82dc-4527-a896-20104fe5508d" (UID: "d4073f0c-82dc-4527-a896-20104fe5508d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:36.909492 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.891466 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt" pod "d4073f0c-82dc-4527-a896-20104fe5508d" (UID: "d4073f0c-82dc-4527-a896-20104fe5508d"). InnerVolumeSpecName "kube-api-access-2cdnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:36.909492 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.900472 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls" pod "d4073f0c-82dc-4527-a896-20104fe5508d" (UID: "d4073f0c-82dc-4527-a896-20104fe5508d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:36.909492 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.904284 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs" pod "d4073f0c-82dc-4527-a896-20104fe5508d" (UID: "d4073f0c-82dc-4527-a896-20104fe5508d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:36.993186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.988391 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:36.993186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.988493 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4073f0c-82dc-4527-a896-20104fe5508d-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:36.993186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.988510 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdnt\" (UniqueName: \"kubernetes.io/projected/d4073f0c-82dc-4527-a896-20104fe5508d-kube-api-access-2cdnt\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:36.993186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:36.988526 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4073f0c-82dc-4527-a896-20104fe5508d-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:37.728131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:37.727864 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" Oct 30 05:52:37.734129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:37.734077 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr" event={"ID":"d4073f0c-82dc-4527-a896-20104fe5508d","Type":"ContainerDied","Data":"a7bcbd1137834a14e543eff3a1c0fd15c07b91a2075a3f729e0f4309110106a0"} Oct 30 05:52:37.734482 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:37.734464 5980 scope.go:122] "RemoveContainer" containerID="8610f5bd6585958caddbc27b64f44fa08910955f200f874eeedb812575cec14f" Oct 30 05:52:37.796809 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:37.796774 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr"] Oct 30 05:52:37.807584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:37.807553 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-54d4943b-5774996df6-zwkbr"] Oct 30 05:52:38.014309 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:38.014236 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4073f0c-82dc-4527-a896-20104fe5508d" path="/var/lib/kubelet/pods/d4073f0c-82dc-4527-a896-20104fe5508d/volumes" Oct 30 05:52:39.616882 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:39.616833 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" podUID="d2162fae-53cf-45e5-84aa-cd08baefde3f" containerName="payload-processing" containerID="cri-o://16eb2f3dc801ffee82bb4f2efb36c5f2eb0bef1b570a220e82fce8e304eb119f" gracePeriod=25 Oct 30 05:52:39.658331 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:39.658292 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" podUID="38a6af8b-ac86-4edb-be94-fc9a35ebaa39" containerName="payload-pre-processing" containerID="cri-o://86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421" gracePeriod=25 Oct 30 05:52:39.750992 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:39.750951 5980 generic.go:356] "Generic (PLEG): container finished" podID="d2162fae-53cf-45e5-84aa-cd08baefde3f" containerID="16eb2f3dc801ffee82bb4f2efb36c5f2eb0bef1b570a220e82fce8e304eb119f" exitCode=0 Oct 30 05:52:39.751140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:39.751066 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" event={"ID":"d2162fae-53cf-45e5-84aa-cd08baefde3f","Type":"ContainerDied","Data":"16eb2f3dc801ffee82bb4f2efb36c5f2eb0bef1b570a220e82fce8e304eb119f"} Oct 30 05:52:40.079997 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.079700 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:52:40.145376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.144831 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7\" (UniqueName: \"kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7\") pod \"d2162fae-53cf-45e5-84aa-cd08baefde3f\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " Oct 30 05:52:40.145376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.144972 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume\") pod \"d2162fae-53cf-45e5-84aa-cd08baefde3f\" (UID: \"d2162fae-53cf-45e5-84aa-cd08baefde3f\") " Oct 30 05:52:40.146469 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.146434 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume" pod "d2162fae-53cf-45e5-84aa-cd08baefde3f" (UID: "d2162fae-53cf-45e5-84aa-cd08baefde3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:40.146840 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.146814 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2162fae-53cf-45e5-84aa-cd08baefde3f-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.201172 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.201042 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7" pod "d2162fae-53cf-45e5-84aa-cd08baefde3f" (UID: "d2162fae-53cf-45e5-84aa-cd08baefde3f"). InnerVolumeSpecName "kube-api-access-7ctq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:40.268188 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.265572 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctq7\" (UniqueName: \"kubernetes.io/projected/d2162fae-53cf-45e5-84aa-cd08baefde3f-kube-api-access-7ctq7\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.308452 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.308415 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:52:40.367087 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.367053 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt\" (UniqueName: \"kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt\") pod \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " Oct 30 05:52:40.369575 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.369545 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt" pod "38a6af8b-ac86-4edb-be94-fc9a35ebaa39" (UID: "38a6af8b-ac86-4edb-be94-fc9a35ebaa39"). InnerVolumeSpecName "kube-api-access-pvjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:40.468821 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.468723 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume\") pod \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\" (UID: \"38a6af8b-ac86-4edb-be94-fc9a35ebaa39\") " Oct 30 05:52:40.469849 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.469072 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume" pod "38a6af8b-ac86-4edb-be94-fc9a35ebaa39" (UID: "38a6af8b-ac86-4edb-be94-fc9a35ebaa39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:40.469849 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.469629 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.469849 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.469653 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjvt\" (UniqueName: \"kubernetes.io/projected/38a6af8b-ac86-4edb-be94-fc9a35ebaa39-kube-api-access-pvjvt\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.759687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.759592 5980 generic.go:356] "Generic (PLEG): container finished" podID="38a6af8b-ac86-4edb-be94-fc9a35ebaa39" containerID="86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421" exitCode=0 Oct 30 05:52:40.759687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.759619 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" event={"ID":"38a6af8b-ac86-4edb-be94-fc9a35ebaa39","Type":"ContainerDied","Data":"86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421"} Oct 30 05:52:40.759687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.759668 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" Oct 30 05:52:40.760297 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.759715 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn" event={"ID":"38a6af8b-ac86-4edb-be94-fc9a35ebaa39","Type":"ContainerDied","Data":"8af6518fe9907eb383b9ad97950452fcd0c4b14e9b238b72eba83ca16862ab79"} Oct 30 05:52:40.760297 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.759746 5980 scope.go:122] "RemoveContainer" containerID="86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421" Oct 30 05:52:40.763625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.761955 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" event={"ID":"d2162fae-53cf-45e5-84aa-cd08baefde3f","Type":"ContainerDied","Data":"3271f09963a69db67e8756760fbe14e6ecf002d6dc8692236f4c432bb61c512d"} Oct 30 05:52:40.763625 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.762000 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd" Oct 30 05:52:40.772046 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.772030 5980 scope.go:122] "RemoveContainer" containerID="86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421" Oct 30 05:52:40.772381 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:52:40.772305 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421\": container with ID starting with 86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421 not found: ID does not exist" containerID="86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421" Oct 30 05:52:40.772381 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.772329 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421"} err="failed to get container status \"86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421\": rpc error: code = NotFound desc = could not find container \"86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421\": container with ID starting with 86ec97e1920fc0ea4c1b8e9f6e1e07e01ff0945b14facf903868dbde73a3e421 not found: ID does not exist" Oct 30 05:52:40.772381 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.772347 5980 scope.go:122] "RemoveContainer" containerID="16eb2f3dc801ffee82bb4f2efb36c5f2eb0bef1b570a220e82fce8e304eb119f" Oct 30 05:52:40.836496 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.836469 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn"] Oct 30 05:52:40.847327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.847304 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-709c4831-d78d7899f-d4hmn"] Oct 30 05:52:40.873432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.873396 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd"] Oct 30 05:52:40.878665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:40.878638 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-709c4831-7f7c6c9465-kb4qd"] Oct 30 05:52:42.014672 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:42.014641 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a6af8b-ac86-4edb-be94-fc9a35ebaa39" path="/var/lib/kubelet/pods/38a6af8b-ac86-4edb-be94-fc9a35ebaa39/volumes" Oct 30 05:52:42.015051 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:42.015011 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2162fae-53cf-45e5-84aa-cd08baefde3f" path="/var/lib/kubelet/pods/d2162fae-53cf-45e5-84aa-cd08baefde3f/volumes" Oct 30 05:52:43.088400 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:52:43.088362 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:52:43.743937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:43.743910 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 05:52:47.106251 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.106213 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg"] Oct 30 05:52:47.107030 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107010 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d4073f0c-82dc-4527-a896-20104fe5508d" containerName="maas-api" Oct 30 05:52:47.107082 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107034 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4073f0c-82dc-4527-a896-20104fe5508d" containerName="maas-api" Oct 30 05:52:47.107144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107087 5980 cpu_manager.go:414] "containerMap: removing container" podUID="38a6af8b-ac86-4edb-be94-fc9a35ebaa39" containerName="payload-pre-processing" Oct 30 05:52:47.107144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107095 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a6af8b-ac86-4edb-be94-fc9a35ebaa39" containerName="payload-pre-processing" Oct 30 05:52:47.107144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107135 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d2162fae-53cf-45e5-84aa-cd08baefde3f" containerName="payload-processing" Oct 30 05:52:47.107144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107143 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2162fae-53cf-45e5-84aa-cd08baefde3f" containerName="payload-processing" Oct 30 05:52:47.107347 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107330 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a6af8b-ac86-4edb-be94-fc9a35ebaa39" containerName="payload-pre-processing" Oct 30 05:52:47.107384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107354 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2162fae-53cf-45e5-84aa-cd08baefde3f" containerName="payload-processing" Oct 30 05:52:47.107384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.107369 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4073f0c-82dc-4527-a896-20104fe5508d" containerName="maas-api" Oct 30 05:52:47.111572 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.111553 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.141467 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.141441 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e2a1565f\"" Oct 30 05:52:47.154437 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.154407 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e2a1565f-dockercfg-8czxk\"" Oct 30 05:52:47.159123 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.159078 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg"] Oct 30 05:52:47.160991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.160967 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k"] Oct 30 05:52:47.165528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.165508 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.188093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.188063 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k"] Oct 30 05:52:47.226283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.226248 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.226934 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.226547 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjnf\" (UniqueName: \"kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf\") pod \"payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.333921 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.333885 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.334186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.333964 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume\") pod \"payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.334186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.333999 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjnf\" (UniqueName: \"kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf\") pod \"payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.336480 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.335024 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.336480 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.335529 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjr6\" (UniqueName: \"kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6\") pod \"payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.441432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.435783 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjnf\" (UniqueName: \"kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf\") pod \"payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.441432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.437022 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjr6\" (UniqueName: \"kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6\") pod \"payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.441432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.437313 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume\") pod \"payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.441432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.438093 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume\") pod \"payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.462863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.462831 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjr6\" (UniqueName: \"kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6\") pod \"payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.516983 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.516947 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:47.728184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.726054 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:47.883003 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:47.882973 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k"] Oct 30 05:52:47.883166 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:52:47.883009 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8d9dd2_276f_40a8_bed2_2c815e95682e.slice/crio-c6dea9a3b05aa8e3c4b34f58e0c30067ef4150941d3a98d677c6cdbce25f3894 WatchSource:0}: Error finding container c6dea9a3b05aa8e3c4b34f58e0c30067ef4150941d3a98d677c6cdbce25f3894: Status 404 returned error can't find the container with id c6dea9a3b05aa8e3c4b34f58e0c30067ef4150941d3a98d677c6cdbce25f3894 Oct 30 05:52:48.056333 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.056306 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg"] Oct 30 05:52:48.057450 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:52:48.057427 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d17981_a5f4_499a_856b_965e30bbe1c0.slice/crio-65df503db8dbedb3801de56e0699a2c404120a32759be558b378cf83ab2ec2ef WatchSource:0}: Error finding container 65df503db8dbedb3801de56e0699a2c404120a32759be558b378cf83ab2ec2ef: Status 404 returned error can't find the container with id 65df503db8dbedb3801de56e0699a2c404120a32759be558b378cf83ab2ec2ef Oct 30 05:52:48.842990 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.842950 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" event={"ID":"98d17981-a5f4-499a-856b-965e30bbe1c0","Type":"ContainerStarted","Data":"b0c02209b3dd0783c4cc57c4203cd6eea9e10c6f2b2eb6738fed5fffce73fe1d"} Oct 30 05:52:48.843438 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.843014 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" event={"ID":"98d17981-a5f4-499a-856b-965e30bbe1c0","Type":"ContainerStarted","Data":"65df503db8dbedb3801de56e0699a2c404120a32759be558b378cf83ab2ec2ef"} Oct 30 05:52:48.843438 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.843036 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:48.845203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.845176 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" event={"ID":"fe8d9dd2-276f-40a8-bed2-2c815e95682e","Type":"ContainerStarted","Data":"42765ffb66860134fb2ffba964696fdf89ec9ccfb234f2be1b222a80971ed47e"} Oct 30 05:52:48.845330 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.845225 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" event={"ID":"fe8d9dd2-276f-40a8-bed2-2c815e95682e","Type":"ContainerStarted","Data":"c6dea9a3b05aa8e3c4b34f58e0c30067ef4150941d3a98d677c6cdbce25f3894"} Oct 30 05:52:48.845330 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.845246 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:48.887050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.887005 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" podStartSLOduration=1.886992209 podStartE2EDuration="1.886992209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:48.883989147 +0000 UTC m=+1222.992028179" watchObservedRunningTime="2026-10-30 05:52:48.886992209 +0000 UTC m=+1222.995031258" Oct 30 05:52:48.930151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:48.930095 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" podStartSLOduration=1.93008207 podStartE2EDuration="1.93008207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:48.927034738 +0000 UTC m=+1223.035073770" watchObservedRunningTime="2026-10-30 05:52:48.93008207 +0000 UTC m=+1223.038121103" Oct 30 05:52:58.434220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.434186 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99"] Oct 30 05:52:58.438686 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.438670 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.481756 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.481918 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8pk\" (UniqueName: \"kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.482141 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.482188 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.492131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.482751 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a095fc35-serving-cert\"" Oct 30 05:52:58.511022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.510943 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99"] Oct 30 05:52:58.588171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.586140 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8pk\" (UniqueName: \"kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.588171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.586350 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.588171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.586394 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.588171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.586502 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.588171 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.587376 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.591582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.591559 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.591866 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.591848 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.625927 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.625871 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8pk\" (UniqueName: \"kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk\") pod \"maas-api-e2e-ait-a095fc35-746c6b9744-8fw99\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:58.752957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:58.752879 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:52:59.161092 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:59.161062 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99"] Oct 30 05:52:59.162094 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:52:59.162053 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f378359_3993_4169_b9c7_9806cf489a62.slice/crio-654e9431753fb9bf8f495050d0f1444dbf356aa72c1e3feb070959ff9817e670 WatchSource:0}: Error finding container 654e9431753fb9bf8f495050d0f1444dbf356aa72c1e3feb070959ff9817e670: Status 404 returned error can't find the container with id 654e9431753fb9bf8f495050d0f1444dbf356aa72c1e3feb070959ff9817e670 Oct 30 05:52:59.881977 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:59.879667 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:52:59.883753 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:59.883281 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:52:59.976343 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:59.976304 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" event={"ID":"4f378359-3993-4169-b9c7-9806cf489a62","Type":"ContainerStarted","Data":"3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7"} Oct 30 05:52:59.976529 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:59.976372 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" event={"ID":"4f378359-3993-4169-b9c7-9806cf489a62","Type":"ContainerStarted","Data":"654e9431753fb9bf8f495050d0f1444dbf356aa72c1e3feb070959ff9817e670"} Oct 30 05:52:59.976716 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:52:59.976663 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:53:02.103559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.103501 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" podStartSLOduration=3.7668399409999997 podStartE2EDuration="4.10348446s" totalImagesPullingTime="336.644519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:00.033425808 +0000 UTC m=+1234.141464841" watchObservedRunningTime="2026-10-30 05:53:02.10348446 +0000 UTC m=+1236.211523511" Oct 30 05:53:02.104983 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.104964 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2"] Oct 30 05:53:02.120969 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.120950 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.132788 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.132749 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2"] Oct 30 05:53:02.142155 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.139634 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e2a1565f-serving-cert\"" Oct 30 05:53:02.208129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.208086 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.208316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.208236 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.208316 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.208305 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.208559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.208538 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsx6f\" (UniqueName: \"kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.310204 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.310176 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.310358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.310242 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.310466 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.310434 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsx6f\" (UniqueName: \"kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.310600 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.310584 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.310963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.310937 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.312952 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.312925 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.313453 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.313430 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.336241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.336219 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsx6f\" (UniqueName: \"kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f\") pod \"maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.470143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.470021 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:02.938420 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:02.937343 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2"] Oct 30 05:53:03.071267 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:03.071171 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" event={"ID":"68be8122-a0b1-42e7-8041-eaa9ddd1437e","Type":"ContainerStarted","Data":"3a074c7a64d9c048a4a02f6c720ab6480ebece780f5a8571ae0c450c712c9ae7"} Oct 30 05:53:04.085554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:04.083722 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" event={"ID":"68be8122-a0b1-42e7-8041-eaa9ddd1437e","Type":"ContainerStarted","Data":"9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658"} Oct 30 05:53:04.085554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:04.083891 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:04.130968 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:04.130899 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" podStartSLOduration=1.47811871 podStartE2EDuration="2.130878326s" totalImagesPullingTime="652.759616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:04.127189257 +0000 UTC m=+1238.235228297" watchObservedRunningTime="2026-10-30 05:53:04.130878326 +0000 UTC m=+1238.238917377" Oct 30 05:53:05.094763 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:05.094719 5980 generic.go:356] "Generic (PLEG): container finished" podID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerID="6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2" exitCode=0 Oct 30 05:53:05.095279 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:05.094787 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" event={"ID":"a274fed5-ff91-44aa-a19f-129c6b9c8b32","Type":"ContainerDied","Data":"6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2"} Oct 30 05:53:05.095665 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:05.095646 5980 scope.go:122] "RemoveContainer" containerID="6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2" Oct 30 05:53:05.097397 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:05.097361 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:05.097519 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:05.097450 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:05.097519 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:05.097495 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:06.108400 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:06.108366 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" event={"ID":"a274fed5-ff91-44aa-a19f-129c6b9c8b32","Type":"ContainerStarted","Data":"16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0"} Oct 30 05:53:06.297128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:06.297072 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:53:06.297431 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:06.297404 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:53:06.297487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:06.297468 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 05:53:07.248395 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:07.238538 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:53:07.310133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:07.299683 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:53:07.310133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:07.299786 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 05:53:08.300580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:08.300538 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:53:08.301210 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:08.300602 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 05:53:09.296755 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:09.296720 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:53:09.296975 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:09.296816 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 05:53:10.106128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:10.106084 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:10.311134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:10.306681 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:53:10.311134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:10.306745 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 05:53:11.297263 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:11.297228 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 05:53:11.297691 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:11.297293 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 05:53:12.302335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:12.302307 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:53:12.302827 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:12.302721 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:53:12.303431 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:12.303412 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 05:53:13.105002 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:53:13.104967 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:53:23.050129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.049278 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q"] Oct 30 05:53:23.054338 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.054315 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.082832 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.082796 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.083476 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.083452 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.083678 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.083664 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv7mc\" (UniqueName: \"kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.083868 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.083856 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.101913 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.098780 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q"] Oct 30 05:53:23.165402 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.165355 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-c73fcb-serving-cert\"" Oct 30 05:53:23.185462 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.185434 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.185699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.185682 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv7mc\" (UniqueName: \"kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.185794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.185785 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.185897 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.185888 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.186296 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:53:23.186272 5980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-serving-cert: secret "maas-api-e2e-shared-b-w3-c73fcb-serving-cert" not found Oct 30 05:53:23.186489 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:53:23.186477 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls podName:5fce3f95-3101-4a0b-9bed-1b4fafef341c nodeName:}" failed. No retries permitted until 2026-10-30 05:53:23.686455646 +0000 UTC m=+1257.794494678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls") pod "maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" (UID: "5fce3f95-3101-4a0b-9bed-1b4fafef341c") : secret "maas-api-e2e-shared-b-w3-c73fcb-serving-cert" not found Oct 30 05:53:23.186838 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.186815 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.192650 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.192625 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.215221 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.215189 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv7mc\" (UniqueName: \"kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.695657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.695594 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.699247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.699219 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:23.746504 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:23.746469 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:24.187087 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:24.187056 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q"] Oct 30 05:53:24.189485 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:53:24.189459 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fce3f95_3101_4a0b_9bed_1b4fafef341c.slice/crio-27af4c8d62d6501c918505acc8479acbb4405bdb6c2692fae979ed0b9402027f WatchSource:0}: Error finding container 27af4c8d62d6501c918505acc8479acbb4405bdb6c2692fae979ed0b9402027f: Status 404 returned error can't find the container with id 27af4c8d62d6501c918505acc8479acbb4405bdb6c2692fae979ed0b9402027f Oct 30 05:53:24.594410 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:24.594371 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" event={"ID":"5fce3f95-3101-4a0b-9bed-1b4fafef341c","Type":"ContainerStarted","Data":"27af4c8d62d6501c918505acc8479acbb4405bdb6c2692fae979ed0b9402027f"} Oct 30 05:53:25.604291 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:25.604246 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" event={"ID":"5fce3f95-3101-4a0b-9bed-1b4fafef341c","Type":"ContainerStarted","Data":"7b7c6e630d33aaa0da0366934a136b42670d21ee339f55111287ceedc3c0f445"} Oct 30 05:53:25.604802 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:25.604324 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:25.655881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:25.655827 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" podStartSLOduration=2.323335561 podStartE2EDuration="2.655814377s" totalImagesPullingTime="332.478816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:25.653668979 +0000 UTC m=+1259.761708013" watchObservedRunningTime="2026-10-30 05:53:25.655814377 +0000 UTC m=+1259.763853410" Oct 30 05:53:31.630093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:31.630059 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 05:53:36.480462 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:36.480375 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k"] Oct 30 05:53:36.501208 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:36.501176 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg"] Oct 30 05:53:38.443779 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.443746 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh"] Oct 30 05:53:38.447863 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.447844 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:38.465984 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.465959 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1b2243ef-dockercfg-q8xbf\"" Oct 30 05:53:38.466097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.466014 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1b2243ef\"" Oct 30 05:53:38.765974 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.765898 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh"] Oct 30 05:53:38.788830 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.788734 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm"] Oct 30 05:53:38.800731 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.800703 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm"] Oct 30 05:53:38.801038 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.800984 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:38.867164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.867094 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6krn\" (UniqueName: \"kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:38.867359 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.867184 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:38.867359 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.867223 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:38.867485 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.867419 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pjd\" (UniqueName: \"kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:38.968466 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.968432 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pjd\" (UniqueName: \"kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:38.968667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.968567 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6krn\" (UniqueName: \"kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:38.968667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.968612 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:38.968667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.968645 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:38.969593 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.969566 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:38.970375 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.970355 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:38.996463 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:38.996442 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pjd\" (UniqueName: \"kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:39.001346 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.001299 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6krn\" (UniqueName: \"kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:39.060971 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.060869 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:39.123591 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.123559 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:39.397788 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.397760 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh"] Oct 30 05:53:39.399205 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:53:39.399182 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56d04d5_c6d9_496a_adf2_9417b87f800b.slice/crio-5f30d9246c4db5983145d1fc8b5059eb127a36a791d706eb0497e50690d4efa0 WatchSource:0}: Error finding container 5f30d9246c4db5983145d1fc8b5059eb127a36a791d706eb0497e50690d4efa0: Status 404 returned error can't find the container with id 5f30d9246c4db5983145d1fc8b5059eb127a36a791d706eb0497e50690d4efa0 Oct 30 05:53:39.504949 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.504918 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm"] Oct 30 05:53:39.756745 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.756653 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" event={"ID":"b56d04d5-c6d9-496a-adf2-9417b87f800b","Type":"ContainerStarted","Data":"d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d"} Oct 30 05:53:39.757003 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.756980 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" event={"ID":"b56d04d5-c6d9-496a-adf2-9417b87f800b","Type":"ContainerStarted","Data":"5f30d9246c4db5983145d1fc8b5059eb127a36a791d706eb0497e50690d4efa0"} Oct 30 05:53:39.758287 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.758268 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:53:39.760182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.760161 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" event={"ID":"45bd9927-e42c-4a6c-8940-7c090db30a62","Type":"ContainerStarted","Data":"14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8"} Oct 30 05:53:39.760360 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.760341 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" event={"ID":"45bd9927-e42c-4a6c-8940-7c090db30a62","Type":"ContainerStarted","Data":"bd2ab30ed54d6874d313a4a0c2845a7ec1173edbd5f5e087e777f6fefd40fca5"} Oct 30 05:53:39.760554 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.760539 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:39.884930 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.884872 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" podStartSLOduration=1.884851587 podStartE2EDuration="1.884851587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:39.828385381 +0000 UTC m=+1273.936424414" watchObservedRunningTime="2026-10-30 05:53:39.884851587 +0000 UTC m=+1273.992890621" Oct 30 05:53:39.887901 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:39.887842 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" podStartSLOduration=1.887828936 podStartE2EDuration="1.887828936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:39.883424491 +0000 UTC m=+1273.991463527" watchObservedRunningTime="2026-10-30 05:53:39.887828936 +0000 UTC m=+1273.995867994" Oct 30 05:53:40.803773 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:40.803729 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99"] Oct 30 05:53:40.804313 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:40.803969 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" podUID="4f378359-3993-4169-b9c7-9806cf489a62" containerName="maas-api" containerID="cri-o://3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7" gracePeriod=30 Oct 30 05:53:41.170614 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.170579 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:53:41.279130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.271827 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca\") pod \"4f378359-3993-4169-b9c7-9806cf489a62\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " Oct 30 05:53:41.279130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.271891 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls\") pod \"4f378359-3993-4169-b9c7-9806cf489a62\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " Oct 30 05:53:41.279130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.271981 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs\") pod \"4f378359-3993-4169-b9c7-9806cf489a62\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " Oct 30 05:53:41.279130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.272038 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk\" (UniqueName: \"kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk\") pod \"4f378359-3993-4169-b9c7-9806cf489a62\" (UID: \"4f378359-3993-4169-b9c7-9806cf489a62\") " Oct 30 05:53:41.306451 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.306410 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca" pod "4f378359-3993-4169-b9c7-9806cf489a62" (UID: "4f378359-3993-4169-b9c7-9806cf489a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:41.320095 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.320056 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls" pod "4f378359-3993-4169-b9c7-9806cf489a62" (UID: "4f378359-3993-4169-b9c7-9806cf489a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:41.320355 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.320324 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs" pod "4f378359-3993-4169-b9c7-9806cf489a62" (UID: "4f378359-3993-4169-b9c7-9806cf489a62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:41.321042 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.321009 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk" pod "4f378359-3993-4169-b9c7-9806cf489a62" (UID: "4f378359-3993-4169-b9c7-9806cf489a62"). InnerVolumeSpecName "kube-api-access-kx8pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:41.393335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.393241 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:41.393335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.393276 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f378359-3993-4169-b9c7-9806cf489a62-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:41.393335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.393289 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8pk\" (UniqueName: \"kubernetes.io/projected/4f378359-3993-4169-b9c7-9806cf489a62-kube-api-access-kx8pk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:41.393335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.393302 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f378359-3993-4169-b9c7-9806cf489a62-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:41.480783 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.480720 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" podUID="fe8d9dd2-276f-40a8-bed2-2c815e95682e" containerName="payload-processing" containerID="cri-o://42765ffb66860134fb2ffba964696fdf89ec9ccfb234f2be1b222a80971ed47e" gracePeriod=25 Oct 30 05:53:41.517196 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.511894 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" podUID="98d17981-a5f4-499a-856b-965e30bbe1c0" containerName="payload-pre-processing" containerID="cri-o://b0c02209b3dd0783c4cc57c4203cd6eea9e10c6f2b2eb6738fed5fffce73fe1d" gracePeriod=25 Oct 30 05:53:41.781386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.781140 5980 generic.go:356] "Generic (PLEG): container finished" podID="98d17981-a5f4-499a-856b-965e30bbe1c0" containerID="b0c02209b3dd0783c4cc57c4203cd6eea9e10c6f2b2eb6738fed5fffce73fe1d" exitCode=0 Oct 30 05:53:41.781386 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.781309 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" event={"ID":"98d17981-a5f4-499a-856b-965e30bbe1c0","Type":"ContainerDied","Data":"b0c02209b3dd0783c4cc57c4203cd6eea9e10c6f2b2eb6738fed5fffce73fe1d"} Oct 30 05:53:41.783777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.783749 5980 generic.go:356] "Generic (PLEG): container finished" podID="fe8d9dd2-276f-40a8-bed2-2c815e95682e" containerID="42765ffb66860134fb2ffba964696fdf89ec9ccfb234f2be1b222a80971ed47e" exitCode=0 Oct 30 05:53:41.784241 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.783961 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" event={"ID":"fe8d9dd2-276f-40a8-bed2-2c815e95682e","Type":"ContainerDied","Data":"42765ffb66860134fb2ffba964696fdf89ec9ccfb234f2be1b222a80971ed47e"} Oct 30 05:53:41.786280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.785968 5980 generic.go:356] "Generic (PLEG): container finished" podID="4f378359-3993-4169-b9c7-9806cf489a62" containerID="3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7" exitCode=0 Oct 30 05:53:41.801774 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.799954 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" Oct 30 05:53:41.820022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.810498 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" event={"ID":"4f378359-3993-4169-b9c7-9806cf489a62","Type":"ContainerDied","Data":"3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7"} Oct 30 05:53:41.822019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.820635 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99" event={"ID":"4f378359-3993-4169-b9c7-9806cf489a62","Type":"ContainerDied","Data":"654e9431753fb9bf8f495050d0f1444dbf356aa72c1e3feb070959ff9817e670"} Oct 30 05:53:41.822019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.820729 5980 scope.go:122] "RemoveContainer" containerID="3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7" Oct 30 05:53:41.867875 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.867814 5980 scope.go:122] "RemoveContainer" containerID="3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7" Oct 30 05:53:41.869061 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:53:41.868886 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7\": container with ID starting with 3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7 not found: ID does not exist" containerID="3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7" Oct 30 05:53:41.869239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.869208 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7"} err="failed to get container status \"3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7\": rpc error: code = NotFound desc = could not find container \"3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7\": container with ID starting with 3af74c5d7e7ebfc602f4b87d508adf75b8a5c6551e92377f9c181b4647a0afc7 not found: ID does not exist" Oct 30 05:53:41.931180 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.920402 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99"] Oct 30 05:53:41.931180 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.925288 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a095fc35-746c6b9744-8fw99"] Oct 30 05:53:41.950700 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:41.950670 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:53:42.042202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.039244 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f378359-3993-4169-b9c7-9806cf489a62" path="/var/lib/kubelet/pods/4f378359-3993-4169-b9c7-9806cf489a62/volumes" Oct 30 05:53:42.042202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.040350 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6\" (UniqueName: \"kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6\") pod \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " Oct 30 05:53:42.042202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.040677 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume\") pod \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\" (UID: \"fe8d9dd2-276f-40a8-bed2-2c815e95682e\") " Oct 30 05:53:42.042202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.041767 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume" pod "fe8d9dd2-276f-40a8-bed2-2c815e95682e" (UID: "fe8d9dd2-276f-40a8-bed2-2c815e95682e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:42.055612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.055565 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6" pod "fe8d9dd2-276f-40a8-bed2-2c815e95682e" (UID: "fe8d9dd2-276f-40a8-bed2-2c815e95682e"). InnerVolumeSpecName "kube-api-access-6xjr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:42.079481 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.079455 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:53:42.142882 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.142851 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume\") pod \"98d17981-a5f4-499a-856b-965e30bbe1c0\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " Oct 30 05:53:42.143046 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.142944 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf\" (UniqueName: \"kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf\") pod \"98d17981-a5f4-499a-856b-965e30bbe1c0\" (UID: \"98d17981-a5f4-499a-856b-965e30bbe1c0\") " Oct 30 05:53:42.143242 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.143219 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume" pod "98d17981-a5f4-499a-856b-965e30bbe1c0" (UID: "98d17981-a5f4-499a-856b-965e30bbe1c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:42.143667 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.143647 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe8d9dd2-276f-40a8-bed2-2c815e95682e-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:42.143752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.143673 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98d17981-a5f4-499a-856b-965e30bbe1c0-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:42.143752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.143686 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjr6\" (UniqueName: \"kubernetes.io/projected/fe8d9dd2-276f-40a8-bed2-2c815e95682e-kube-api-access-6xjr6\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:42.145407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.145385 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf" pod "98d17981-a5f4-499a-856b-965e30bbe1c0" (UID: "98d17981-a5f4-499a-856b-965e30bbe1c0"). InnerVolumeSpecName "kube-api-access-lzjnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:42.245045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.244965 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjnf\" (UniqueName: \"kubernetes.io/projected/98d17981-a5f4-499a-856b-965e30bbe1c0-kube-api-access-lzjnf\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:42.797117 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.797084 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" Oct 30 05:53:42.797300 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.797100 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg" event={"ID":"98d17981-a5f4-499a-856b-965e30bbe1c0","Type":"ContainerDied","Data":"65df503db8dbedb3801de56e0699a2c404120a32759be558b378cf83ab2ec2ef"} Oct 30 05:53:42.797409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.797378 5980 scope.go:122] "RemoveContainer" containerID="b0c02209b3dd0783c4cc57c4203cd6eea9e10c6f2b2eb6738fed5fffce73fe1d" Oct 30 05:53:42.801074 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.801044 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" event={"ID":"fe8d9dd2-276f-40a8-bed2-2c815e95682e","Type":"ContainerDied","Data":"c6dea9a3b05aa8e3c4b34f58e0c30067ef4150941d3a98d677c6cdbce25f3894"} Oct 30 05:53:42.801255 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.801085 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k" Oct 30 05:53:42.813470 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.813451 5980 scope.go:122] "RemoveContainer" containerID="42765ffb66860134fb2ffba964696fdf89ec9ccfb234f2be1b222a80971ed47e" Oct 30 05:53:42.864718 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.864684 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k"] Oct 30 05:53:42.876234 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.876210 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e2a1565f-6c755fd957-gh74k"] Oct 30 05:53:42.905622 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.905595 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg"] Oct 30 05:53:42.919287 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:42.919267 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e2a1565f-bbd67fcc9-z6fvg"] Oct 30 05:53:43.090051 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:53:43.090013 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:53:43.913995 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:43.913962 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2"] Oct 30 05:53:43.914409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:43.914221 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" podUID="68be8122-a0b1-42e7-8041-eaa9ddd1437e" containerName="maas-api" containerID="cri-o://9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658" gracePeriod=30 Oct 30 05:53:44.018277 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.018239 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d17981-a5f4-499a-856b-965e30bbe1c0" path="/var/lib/kubelet/pods/98d17981-a5f4-499a-856b-965e30bbe1c0/volumes" Oct 30 05:53:44.018723 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.018705 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8d9dd2-276f-40a8-bed2-2c815e95682e" path="/var/lib/kubelet/pods/fe8d9dd2-276f-40a8-bed2-2c815e95682e/volumes" Oct 30 05:53:44.201871 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.201844 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:44.213235 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.213210 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs\") pod \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " Oct 30 05:53:44.213377 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.213363 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls\") pod \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " Oct 30 05:53:44.213444 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.213416 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca\") pod \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " Oct 30 05:53:44.213497 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.213466 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f\" (UniqueName: \"kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f\") pod \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\" (UID: \"68be8122-a0b1-42e7-8041-eaa9ddd1437e\") " Oct 30 05:53:44.213823 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.213793 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca" pod "68be8122-a0b1-42e7-8041-eaa9ddd1437e" (UID: "68be8122-a0b1-42e7-8041-eaa9ddd1437e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:44.214434 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.214409 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68be8122-a0b1-42e7-8041-eaa9ddd1437e-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:44.215535 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.215507 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs" pod "68be8122-a0b1-42e7-8041-eaa9ddd1437e" (UID: "68be8122-a0b1-42e7-8041-eaa9ddd1437e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:44.215924 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.215906 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f" pod "68be8122-a0b1-42e7-8041-eaa9ddd1437e" (UID: "68be8122-a0b1-42e7-8041-eaa9ddd1437e"). InnerVolumeSpecName "kube-api-access-fsx6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:44.216120 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.216088 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls" pod "68be8122-a0b1-42e7-8041-eaa9ddd1437e" (UID: "68be8122-a0b1-42e7-8041-eaa9ddd1437e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:44.319132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.315711 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:44.319132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.315744 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsx6f\" (UniqueName: \"kubernetes.io/projected/68be8122-a0b1-42e7-8041-eaa9ddd1437e-kube-api-access-fsx6f\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:44.319132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.315758 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68be8122-a0b1-42e7-8041-eaa9ddd1437e-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:53:44.820486 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.820453 5980 generic.go:356] "Generic (PLEG): container finished" podID="68be8122-a0b1-42e7-8041-eaa9ddd1437e" containerID="9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658" exitCode=0 Oct 30 05:53:44.820701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.820505 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" Oct 30 05:53:44.820701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.820507 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" event={"ID":"68be8122-a0b1-42e7-8041-eaa9ddd1437e","Type":"ContainerDied","Data":"9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658"} Oct 30 05:53:44.820701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.820560 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2" event={"ID":"68be8122-a0b1-42e7-8041-eaa9ddd1437e","Type":"ContainerDied","Data":"3a074c7a64d9c048a4a02f6c720ab6480ebece780f5a8571ae0c450c712c9ae7"} Oct 30 05:53:44.820701 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.820589 5980 scope.go:122] "RemoveContainer" containerID="9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658" Oct 30 05:53:44.834780 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.834635 5980 scope.go:122] "RemoveContainer" containerID="9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658" Oct 30 05:53:44.835139 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:53:44.835082 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658\": container with ID starting with 9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658 not found: ID does not exist" containerID="9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658" Oct 30 05:53:44.835228 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.835153 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658"} err="failed to get container status \"9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658\": rpc error: code = NotFound desc = could not find container \"9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658\": container with ID starting with 9e0e943a8d50a1c27b83a34193821380c594dca5dda3f74b944f2a447eb1c658 not found: ID does not exist" Oct 30 05:53:44.909991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.909955 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2"] Oct 30 05:53:44.921375 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.921343 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e2a1565f-694dd5c9fd-tjhf2"] Oct 30 05:53:44.995132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.994812 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r"] Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.995867 5980 cpu_manager.go:414] "containerMap: removing container" podUID="68be8122-a0b1-42e7-8041-eaa9ddd1437e" containerName="maas-api" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.995890 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68be8122-a0b1-42e7-8041-eaa9ddd1437e" containerName="maas-api" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.995914 5980 cpu_manager.go:414] "containerMap: removing container" podUID="4f378359-3993-4169-b9c7-9806cf489a62" containerName="maas-api" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.995922 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f378359-3993-4169-b9c7-9806cf489a62" containerName="maas-api" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.995979 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fe8d9dd2-276f-40a8-bed2-2c815e95682e" containerName="payload-processing" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.995987 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8d9dd2-276f-40a8-bed2-2c815e95682e" containerName="payload-processing" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.996004 5980 cpu_manager.go:414] "containerMap: removing container" podUID="98d17981-a5f4-499a-856b-965e30bbe1c0" containerName="payload-pre-processing" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.996012 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d17981-a5f4-499a-856b-965e30bbe1c0" containerName="payload-pre-processing" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.996229 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8d9dd2-276f-40a8-bed2-2c815e95682e" containerName="payload-processing" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.996258 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d17981-a5f4-499a-856b-965e30bbe1c0" containerName="payload-pre-processing" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.996278 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="68be8122-a0b1-42e7-8041-eaa9ddd1437e" containerName="maas-api" Oct 30 05:53:45.002436 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:44.996289 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f378359-3993-4169-b9c7-9806cf489a62" containerName="maas-api" Oct 30 05:53:45.041154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.036975 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.048367 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.045826 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r"] Oct 30 05:53:45.072656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.072562 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-815776-openshift-default-dockercfg-wzpnm\"" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074604 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074664 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074708 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074756 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074803 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074896 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.074975 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxhk\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.075030 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.077164 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.075086 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185213 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185172 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185390 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185251 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185390 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185358 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185511 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185440 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxhk\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185511 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185498 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185602 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185560 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185729 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185710 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185801 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185762 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.185858 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.185809 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.186393 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.186360 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.187213 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.187179 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.187455 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.187432 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.199052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.197769 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.199052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.198012 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.206242 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.206211 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.228137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.223720 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.256134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.252665 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxhk\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.300722 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.265582 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.391958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.387896 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:45.868131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.864861 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r"] Oct 30 05:53:45.882576 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.882536 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:45.882708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.882655 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:45.882708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:45.882702 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:46.020186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.020156 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68be8122-a0b1-42e7-8041-eaa9ddd1437e" path="/var/lib/kubelet/pods/68be8122-a0b1-42e7-8041-eaa9ddd1437e/volumes" Oct 30 05:53:46.536582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.536549 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr"] Oct 30 05:53:46.541367 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.541349 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.568931 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.568903 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-477f6b\"" Oct 30 05:53:46.569214 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.569176 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-477f6b-dockercfg-z6g67\"" Oct 30 05:53:46.572907 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.572883 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr"] Oct 30 05:53:46.713166 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.701822 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.713166 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.702044 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5625\" (UniqueName: \"kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625\") pod \"payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.735220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.735185 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf"] Oct 30 05:53:46.750239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.750214 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:46.817823 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.817687 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.817991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.817918 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5625\" (UniqueName: \"kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625\") pod \"payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.822078 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.819148 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.827249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.827219 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf"] Oct 30 05:53:46.877150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.875397 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" event={"ID":"7011a201-0f2a-490c-80fe-e9b5cea5398e","Type":"ContainerStarted","Data":"49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8"} Oct 30 05:53:46.877150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.875486 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" event={"ID":"7011a201-0f2a-490c-80fe-e9b5cea5398e","Type":"ContainerStarted","Data":"f0738cb1a70952d7edcbae48c5301a08cbe5289d99ed9375a8d763866b4fa495"} Oct 30 05:53:46.894185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.884788 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5625\" (UniqueName: \"kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625\") pod \"payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:46.919520 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.919463 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:46.919711 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.919543 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8dl\" (UniqueName: \"kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl\") pod \"payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:46.941797 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:46.941746 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podStartSLOduration=2.941731766 podStartE2EDuration="2.941731766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:46.941547995 +0000 UTC m=+1281.049587028" watchObservedRunningTime="2026-10-30 05:53:46.941731766 +0000 UTC m=+1281.049770802" Oct 30 05:53:47.021924 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.021886 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:47.022434 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.021933 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8dl\" (UniqueName: \"kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl\") pod \"payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:47.022778 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.022758 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:47.046981 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.046949 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8dl\" (UniqueName: \"kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl\") pod \"payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:47.156263 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.156226 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:47.175229 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.175202 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:47.254656 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.254454 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v"] Oct 30 05:53:47.282239 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.266729 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.295814 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.294096 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-3571f8-openshift-default-dockercfg-b4fbx\"" Oct 30 05:53:47.295814 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.295705 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v"] Oct 30 05:53:47.397158 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.392971 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.436465 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.436555 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.436627 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.436676 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr5v\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.436864 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.436981 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.437119 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.437156 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.437724 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.437345 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.478631 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.440396 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.575609 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.575705 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.575759 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.575810 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.575843 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr5v\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.575989 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.576084 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.576466 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.577098 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.588539 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.603129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.588821 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.645203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.606810 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.645203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.634730 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.645203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.634785 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.766196 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:53:47.766136 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a48d1d_d63f_4a65_856d_2487f232727f.slice/crio-fa8f5e2c6eecfaa88a8b6a71abccb5ac08365a2b424a5eaa859674dd2c1a796a WatchSource:0}: Error finding container fa8f5e2c6eecfaa88a8b6a71abccb5ac08365a2b424a5eaa859674dd2c1a796a: Status 404 returned error can't find the container with id fa8f5e2c6eecfaa88a8b6a71abccb5ac08365a2b424a5eaa859674dd2c1a796a Oct 30 05:53:47.777405 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.771391 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr"] Oct 30 05:53:47.780584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.779416 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.780584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.780432 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr5v\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.781523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.781469 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.796616 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.796580 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token\") pod \"e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.924154 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.923869 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:47.973684 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.973215 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf"] Oct 30 05:53:48.000168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.998506 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" event={"ID":"07a48d1d-d63f-4a65-856d-2487f232727f","Type":"ContainerStarted","Data":"fa8f5e2c6eecfaa88a8b6a71abccb5ac08365a2b424a5eaa859674dd2c1a796a"} Oct 30 05:53:48.000168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:47.999907 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:48.051783 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:48.051746 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:53:48.379544 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:48.379514 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v"] Oct 30 05:53:48.381434 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:53:48.381407 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96352bb0_9b5b_41d8_b5cb_8877cb92fe80.slice/crio-4900ad1be121c701d9bd7439de5dbfe3fc0035ab2b371386cd40d032d97418c1 WatchSource:0}: Error finding container 4900ad1be121c701d9bd7439de5dbfe3fc0035ab2b371386cd40d032d97418c1: Status 404 returned error can't find the container with id 4900ad1be121c701d9bd7439de5dbfe3fc0035ab2b371386cd40d032d97418c1 Oct 30 05:53:48.384709 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:48.384671 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:48.384813 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:48.384753 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:48.384886 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:48.384808 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:49.012599 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.012563 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" event={"ID":"96352bb0-9b5b-41d8-b5cb-8877cb92fe80","Type":"ContainerStarted","Data":"8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979"} Oct 30 05:53:49.012900 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.012879 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" event={"ID":"96352bb0-9b5b-41d8-b5cb-8877cb92fe80","Type":"ContainerStarted","Data":"4900ad1be121c701d9bd7439de5dbfe3fc0035ab2b371386cd40d032d97418c1"} Oct 30 05:53:49.017604 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.017574 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" event={"ID":"cc0e8018-070f-4add-ab40-569613d1effa","Type":"ContainerStarted","Data":"58d804fdd42dc9f7d874d4d0b952c467a957470d1841f2262e1459d0f663cdf4"} Oct 30 05:53:49.017726 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.017620 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" event={"ID":"cc0e8018-070f-4add-ab40-569613d1effa","Type":"ContainerStarted","Data":"56e00851420e612dd32892f7d44edbb8b8de196f7baa920c6619527f25442cde"} Oct 30 05:53:49.017726 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.017639 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:53:49.020223 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.020197 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" event={"ID":"07a48d1d-d63f-4a65-856d-2487f232727f","Type":"ContainerStarted","Data":"65c7a41c631abc1423bfbda33def343f258ede724881770eefe056f03bf2af20"} Oct 30 05:53:49.021006 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.020989 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:53:49.069163 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.069085 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podStartSLOduration=2.069065401 podStartE2EDuration="2.069065401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:49.060479529 +0000 UTC m=+1283.168518572" watchObservedRunningTime="2026-10-30 05:53:49.069065401 +0000 UTC m=+1283.177104435" Oct 30 05:53:49.106426 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.106371 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" podStartSLOduration=3.106356624 podStartE2EDuration="3.106356624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:49.105084941 +0000 UTC m=+1283.213123975" watchObservedRunningTime="2026-10-30 05:53:49.106356624 +0000 UTC m=+1283.214395658" Oct 30 05:53:49.151789 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.151724 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" podStartSLOduration=3.151703045 podStartE2EDuration="3.151703045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:49.147777014 +0000 UTC m=+1283.255816071" watchObservedRunningTime="2026-10-30 05:53:49.151703045 +0000 UTC m=+1283.259742096" Oct 30 05:53:49.927178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.927145 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:49.932844 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:49.932815 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:50.026534 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:50.026337 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:50.026983 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:50.026966 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:53:50.770977 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:50.770938 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:53:51.825718 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:53:51.825344 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:54:00.029368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:00.029338 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 05:54:00.995125 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:00.995075 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx"] Oct 30 05:54:01.000917 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.000894 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.036718 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.036409 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx"] Oct 30 05:54:01.039045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.039019 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 05:54:01.042300 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.042266 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c73fcb-openshift-default-dockercfg-rtknk\"" Oct 30 05:54:01.045810 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.045786 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7"] Oct 30 05:54:01.051292 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.051273 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.092004 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.091974 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7"] Oct 30 05:54:01.105746 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.105570 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.105746 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.105660 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.105958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.105787 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzpq\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.106676 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.106609 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.107584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.107321 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.107789 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.107767 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.107886 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.107845 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.108877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.108146 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.108877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.108175 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.210897 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.210860 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzdh\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.211096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.210928 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.210954 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211017 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.211096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211049 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211096 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211093 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211410 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211156 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzpq\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211633 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211609 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211698 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211626 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.211698 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211661 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.211803 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211726 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211858 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211816 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211915 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211902 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.211974 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211928 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.212026 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.211988 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.212144 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.212121 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.212219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.212151 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.212274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.212213 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.212274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.212249 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.212479 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.212455 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.212666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.212646 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.213048 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.213024 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.213252 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.213233 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.214398 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.214375 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.214589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.214570 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.239648 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.239593 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.241203 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.241174 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzpq\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.313424 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313342 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313441 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313466 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313517 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313552 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313622 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzdh\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313704 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313813 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.313845 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.313840 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.314422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.314398 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.314422 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.314413 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.314571 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.314542 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.314571 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.314555 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.314849 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.314832 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.316334 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.316312 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.316583 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.316559 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.318741 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.318720 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:01.338725 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.338701 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzdh\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.339737 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.339713 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.372694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.372662 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:01.681456 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:54:01.681408 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84130c06_dc82_437e_a335_5297669035ad.slice/crio-8d4e8a3ce289098cec19303c672a34e7c0076fbfd129313aceb855a82afdb6a5 WatchSource:0}: Error finding container 8d4e8a3ce289098cec19303c672a34e7c0076fbfd129313aceb855a82afdb6a5: Status 404 returned error can't find the container with id 8d4e8a3ce289098cec19303c672a34e7c0076fbfd129313aceb855a82afdb6a5 Oct 30 05:54:01.682642 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.682557 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx"] Oct 30 05:54:01.684814 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.684768 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:01.684932 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.684842 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:01.684932 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.684874 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:01.728711 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.728677 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7"] Oct 30 05:54:01.735868 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:54:01.735800 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648bda6f_2cef_47f5_a7fa_5d571cfa842d.slice/crio-491603d1ad9f6ecf1fd3ad4c75befa49d8fa7402439131212579560c65b6085c WatchSource:0}: Error finding container 491603d1ad9f6ecf1fd3ad4c75befa49d8fa7402439131212579560c65b6085c: Status 404 returned error can't find the container with id 491603d1ad9f6ecf1fd3ad4c75befa49d8fa7402439131212579560c65b6085c Oct 30 05:54:01.741824 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.741785 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:01.741950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.741864 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:01.741950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:01.741892 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:02.226500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.226459 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" event={"ID":"648bda6f-2cef-47f5-a7fa-5d571cfa842d","Type":"ContainerStarted","Data":"1aa064af6d4f2d7eff2d983f25b10449bdfeac613596d8c326018765f9201c2f"} Oct 30 05:54:02.227093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.226561 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" event={"ID":"648bda6f-2cef-47f5-a7fa-5d571cfa842d","Type":"ContainerStarted","Data":"491603d1ad9f6ecf1fd3ad4c75befa49d8fa7402439131212579560c65b6085c"} Oct 30 05:54:02.229039 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.229008 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" event={"ID":"84130c06-dc82-437e-a335-5297669035ad","Type":"ContainerStarted","Data":"98b814a27c6c77b61ff2007927c2fc5b049b2b84d76288f983cf56a6c3b80f4f"} Oct 30 05:54:02.229167 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.229061 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" event={"ID":"84130c06-dc82-437e-a335-5297669035ad","Type":"ContainerStarted","Data":"8d4e8a3ce289098cec19303c672a34e7c0076fbfd129313aceb855a82afdb6a5"} Oct 30 05:54:02.277192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.277128 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podStartSLOduration=2.277092742 podStartE2EDuration="2.277092742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:02.274466733 +0000 UTC m=+1296.382505767" watchObservedRunningTime="2026-10-30 05:54:02.277092742 +0000 UTC m=+1296.385131776" Oct 30 05:54:02.327877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.320638 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:02.327877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.320891 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:54:02.327877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.320938 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 05:54:02.342147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.340512 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podStartSLOduration=2.340493141 podStartE2EDuration="2.340493141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:02.339617144 +0000 UTC m=+1296.447656178" watchObservedRunningTime="2026-10-30 05:54:02.340493141 +0000 UTC m=+1296.448532175" Oct 30 05:54:02.382615 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.382528 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:02.386557 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.386226 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:54:02.386557 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:02.386286 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:54:03.327298 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:03.327266 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:54:03.343686 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:03.343648 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 05:54:03.388133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:03.385647 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:54:03.388133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:03.385708 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:54:04.320179 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:04.320124 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:54:04.320358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:04.320174 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 05:54:04.386143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:04.385631 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:54:04.386143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:04.385695 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:54:05.323219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:05.320455 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:54:05.323219 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:05.320536 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 05:54:05.386131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:05.377671 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:54:05.386131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:05.377744 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:54:06.321481 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:06.321444 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:54:06.321769 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:06.321734 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 05:54:06.381786 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:06.381753 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:54:06.382097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:06.382056 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:54:07.339192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:07.331008 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 05:54:07.339192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:07.331059 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 05:54:07.396717 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:07.386828 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:54:07.396717 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:07.386883 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:54:08.339131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:08.338692 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:08.347153 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:08.339709 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:08.347153 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:08.342530 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:54:08.405057 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:08.404808 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:08.405939 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:08.405910 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:08.407189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:08.407168 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:54:12.237249 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.237218 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w"] Oct 30 05:54:12.242839 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.242820 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.279361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.271821 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-815776-serving-cert\"" Oct 30 05:54:12.279361 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.277219 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w"] Oct 30 05:54:12.345989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.345946 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.346229 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.346200 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.346368 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.346311 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.346588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.346458 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zl2\" (UniqueName: \"kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.448707 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.448675 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.448914 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.448810 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.448985 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.448942 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zl2\" (UniqueName: \"kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.449047 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:54:12.448980 5980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-serving-cert: secret "maas-api-e2e-shared-a-w4-815776-serving-cert" not found Oct 30 05:54:12.449153 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:54:12.449070 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls podName:d8282f0e-56e5-45ca-afa6-db63c4fea78a nodeName:}" failed. No retries permitted until 2026-10-30 05:54:12.949052064 +0000 UTC m=+1307.057091089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls") pod "maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" (UID: "d8282f0e-56e5-45ca-afa6-db63c4fea78a") : secret "maas-api-e2e-shared-a-w4-815776-serving-cert" not found Oct 30 05:54:12.449371 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.449339 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.449500 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.449433 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.452771 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.452744 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.488788 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.480978 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zl2\" (UniqueName: \"kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.958714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.958680 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:12.961420 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:12.961395 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:13.089322 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:54:13.089282 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:54:13.157907 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:13.157877 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:13.524148 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:54:13.524099 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8282f0e_56e5_45ca_afa6_db63c4fea78a.slice/crio-3ca3d4a87d4656afcfdafa83214c65a466c1c8ca1d017ae367fdef6161a7996b WatchSource:0}: Error finding container 3ca3d4a87d4656afcfdafa83214c65a466c1c8ca1d017ae367fdef6161a7996b: Status 404 returned error can't find the container with id 3ca3d4a87d4656afcfdafa83214c65a466c1c8ca1d017ae367fdef6161a7996b Oct 30 05:54:13.524687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:13.524663 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w"] Oct 30 05:54:14.430374 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:14.430274 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" event={"ID":"d8282f0e-56e5-45ca-afa6-db63c4fea78a","Type":"ContainerStarted","Data":"a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6"} Oct 30 05:54:14.430563 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:14.430403 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:14.430563 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:14.430422 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" event={"ID":"d8282f0e-56e5-45ca-afa6-db63c4fea78a","Type":"ContainerStarted","Data":"3ca3d4a87d4656afcfdafa83214c65a466c1c8ca1d017ae367fdef6161a7996b"} Oct 30 05:54:14.487858 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:14.487810 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" podStartSLOduration=2.09697851 podStartE2EDuration="2.487796881s" totalImagesPullingTime="390.818371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:14.477351856 +0000 UTC m=+1308.585390902" watchObservedRunningTime="2026-10-30 05:54:14.487796881 +0000 UTC m=+1308.595835914" Oct 30 05:54:20.442457 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:20.442427 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 05:54:43.090400 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:54:43.090361 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:54:46.836231 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.836197 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72"] Oct 30 05:54:46.843985 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.843950 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.867538 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.867507 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72"] Oct 30 05:54:46.875373 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.875339 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-cc33d3-openshift-default-dockercfg-7lll8\"" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957308 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957609 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957685 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75kl\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957764 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957838 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957871 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.957897 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.958133 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:46.965136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:46.958187 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064032 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064151 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064098 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064215 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064430 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064387 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064543 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064432 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75kl\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064543 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064491 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064649 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064556 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064649 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064605 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.064748 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.064649 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.075135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.066228 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.075135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.066506 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.075135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.066903 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.075135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.068067 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.075135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.069433 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.075135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.075005 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.082280 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.077310 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.103026 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.102914 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.103681 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.103656 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75kl\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.174178 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.173083 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:47.544973 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.544945 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72"] Oct 30 05:54:47.545117 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:54:47.545030 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f345522_495a_42e2_9db8_d07dc01c289a.slice/crio-4ccc44d949c3473e64d382665a54bbaa15aab9d151d45f2245b69fc666f52cb0 WatchSource:0}: Error finding container 4ccc44d949c3473e64d382665a54bbaa15aab9d151d45f2245b69fc666f52cb0: Status 404 returned error can't find the container with id 4ccc44d949c3473e64d382665a54bbaa15aab9d151d45f2245b69fc666f52cb0 Oct 30 05:54:47.549004 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.548970 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:47.549189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.549038 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:47.549189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.549071 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:47.868551 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.868518 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" event={"ID":"8f345522-495a-42e2-9db8-d07dc01c289a","Type":"ContainerStarted","Data":"d197ff44b82e29408bc3a65fb6cc1b8193028bb66fc9efac8316223fe0bff036"} Oct 30 05:54:47.869040 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.869013 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" event={"ID":"8f345522-495a-42e2-9db8-d07dc01c289a","Type":"ContainerStarted","Data":"4ccc44d949c3473e64d382665a54bbaa15aab9d151d45f2245b69fc666f52cb0"} Oct 30 05:54:47.922810 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:47.922748 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podStartSLOduration=1.922726369 podStartE2EDuration="1.922726369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:47.919596461 +0000 UTC m=+1342.027635527" watchObservedRunningTime="2026-10-30 05:54:47.922726369 +0000 UTC m=+1342.030765406" Oct 30 05:54:48.173777 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:48.173696 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:48.175181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:48.175157 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:54:48.175320 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:48.175212 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:54:49.341650 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.340243 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:49.390297 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.389853 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl"] Oct 30 05:54:49.436159 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.436101 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.447247 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.447203 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl"] Oct 30 05:54:49.455495 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.454748 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f22a52-dockercfg-pxgr6\"" Oct 30 05:54:49.456142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.456085 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f22a52\"" Oct 30 05:54:49.465160 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.465129 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvz4\" (UniqueName: \"kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4\") pod \"payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.465751 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.465722 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.578168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.574372 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.578168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.574500 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvz4\" (UniqueName: \"kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4\") pod \"payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.578168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.575822 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.647612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.647532 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvz4\" (UniqueName: \"kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4\") pod \"payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.768405 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.768375 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:49.890388 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.890356 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:49.891336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:49.891315 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:54:50.168385 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:50.168045 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl"] Oct 30 05:54:50.901416 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:50.901369 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" event={"ID":"546784f0-9fd3-4623-b9d7-19923c40b49e","Type":"ContainerStarted","Data":"4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517"} Oct 30 05:54:50.901970 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:50.901457 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" event={"ID":"546784f0-9fd3-4623-b9d7-19923c40b49e","Type":"ContainerStarted","Data":"ff8b856e23824205a0f36db57d16fef10b750a6a70d5e8a599c8d691d3787ea8"} Oct 30 05:54:50.901970 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:50.901864 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:54:50.950796 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:50.950745 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" podStartSLOduration=1.950731242 podStartE2EDuration="1.950731242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:50.950550076 +0000 UTC m=+1345.058589109" watchObservedRunningTime="2026-10-30 05:54:50.950731242 +0000 UTC m=+1345.058770275" Oct 30 05:54:56.902130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:56.900312 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd"] Oct 30 05:54:56.947734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:56.910447 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd"] Oct 30 05:54:56.947734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:56.910565 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:56.947734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:56.940433 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4fc75f\"" Oct 30 05:54:56.947734 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:56.940612 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4fc75f-dockercfg-vrrpt\"" Oct 30 05:54:57.088165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.085493 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxwl\" (UniqueName: \"kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl\") pod \"payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.088165 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.085671 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.192216 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.187299 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxwl\" (UniqueName: \"kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl\") pod \"payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.192216 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.187406 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.192216 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.190666 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.229189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.226955 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxwl\" (UniqueName: \"kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl\") pod \"payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.335820 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.335787 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:57.688509 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:54:57.688472 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a668c6d_8b3d_4fdd_8218_d928cc1462db.slice/crio-79a31a7ad43f87b5bdb69a1ddc6715954dd479976a5c94e9e2e9f352ab6de09d WatchSource:0}: Error finding container 79a31a7ad43f87b5bdb69a1ddc6715954dd479976a5c94e9e2e9f352ab6de09d: Status 404 returned error can't find the container with id 79a31a7ad43f87b5bdb69a1ddc6715954dd479976a5c94e9e2e9f352ab6de09d Oct 30 05:54:57.690052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.690014 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd"] Oct 30 05:54:57.982676 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.982574 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" event={"ID":"8a668c6d-8b3d-4fdd-8218-d928cc1462db","Type":"ContainerStarted","Data":"bc7c8da89823ea6f390781e46b0c1e6465852bda305fbd43c98aa1c2fb5852d9"} Oct 30 05:54:57.982676 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.982659 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" event={"ID":"8a668c6d-8b3d-4fdd-8218-d928cc1462db","Type":"ContainerStarted","Data":"79a31a7ad43f87b5bdb69a1ddc6715954dd479976a5c94e9e2e9f352ab6de09d"} Oct 30 05:54:57.983976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:57.983875 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:54:58.032147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:54:58.032072 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" podStartSLOduration=2.032055842 podStartE2EDuration="2.032055842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:58.028653215 +0000 UTC m=+1352.136692265" watchObservedRunningTime="2026-10-30 05:54:58.032055842 +0000 UTC m=+1352.140094875" Oct 30 05:55:01.916680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:01.916652 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 05:55:10.029984 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.029926 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:55:10.114492 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.114458 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw"] Oct 30 05:55:10.120464 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.120443 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.148971 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.148863 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw"] Oct 30 05:55:10.157152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.157124 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b202a0-openshift-default-dockercfg-kj84r\"" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295296 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295350 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295391 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295448 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4rg\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295539 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295603 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295684 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295717 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.307407 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.295772 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.397934 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4rg\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398045 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398131 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398212 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398243 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398297 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398537 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398566 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.398680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.398610 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.424708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.400706 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.424708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.401396 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.424708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.401617 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.424708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.402272 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.424708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.404627 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.481588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.427898 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.481588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.455739 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4rg\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.492933 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.492856 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.495842 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.495784 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo\") pod \"e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:10.750862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:10.750808 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:11.152147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:11.152121 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw"] Oct 30 05:55:11.152977 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:55:11.152952 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aebf5b9_d569_40b4_8dad_672fbf43ff17.slice/crio-4765d0a53563e2e55b76e3e56753e8e85b75528e1f708627baf1441089187303 WatchSource:0}: Error finding container 4765d0a53563e2e55b76e3e56753e8e85b75528e1f708627baf1441089187303: Status 404 returned error can't find the container with id 4765d0a53563e2e55b76e3e56753e8e85b75528e1f708627baf1441089187303 Oct 30 05:55:11.156530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:11.156497 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:11.156623 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:11.156565 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:11.156623 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:11.156594 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:11.177071 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:11.176976 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" event={"ID":"1aebf5b9-d569-40b4-8dad-672fbf43ff17","Type":"ContainerStarted","Data":"4765d0a53563e2e55b76e3e56753e8e85b75528e1f708627baf1441089187303"} Oct 30 05:55:12.189098 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:12.189047 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" event={"ID":"1aebf5b9-d569-40b4-8dad-672fbf43ff17","Type":"ContainerStarted","Data":"4b465dca87930121b5f50ff395be4948e92e274ef491db4666f843ffab253002"} Oct 30 05:55:12.250879 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:12.250826 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podStartSLOduration=2.250808861 podStartE2EDuration="2.250808861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:12.247363474 +0000 UTC m=+1366.355402508" watchObservedRunningTime="2026-10-30 05:55:12.250808861 +0000 UTC m=+1366.358847955" Oct 30 05:55:12.751200 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:12.751168 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:12.757528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:12.757504 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:13.113246 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:55:13.113208 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:55:13.200160 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:13.200089 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:13.201263 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:13.201243 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:55:13.899133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:13.893322 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb"] Oct 30 05:55:13.899339 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:13.899183 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:13.928130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:13.923998 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a3efab09-openshift-default-dockercfg-vqhz9\"" Oct 30 05:55:13.936281 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:13.935462 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb"] Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.004633 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.004797 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.004867 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.004904 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.005063 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.005100 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.005160 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.005194 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.014134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.005265 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pd7t\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107193 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107160 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107372 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107224 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107372 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107255 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107405 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107461 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107441 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107553 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107483 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107553 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107516 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107654 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107586 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pd7t\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107710 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107652 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107724 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.107760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.107729 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.108051 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.108027 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.108425 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.108355 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.108592 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.108571 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.110676 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.110653 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.110884 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.110862 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.139319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.133835 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.139319 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.133837 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pd7t\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t\") pod \"e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.214050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.213965 5980 generic.go:356] "Generic (PLEG): container finished" podID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerID="49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8" exitCode=0 Oct 30 05:55:14.215090 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.215063 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" event={"ID":"7011a201-0f2a-490c-80fe-e9b5cea5398e","Type":"ContainerDied","Data":"49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8"} Oct 30 05:55:14.215521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.215502 5980 scope.go:122] "RemoveContainer" containerID="49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8" Oct 30 05:55:14.218524 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.218486 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:14.218670 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.218573 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:14.218670 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.218620 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:14.245190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.238993 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:14.971022 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:14.970980 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb"] Oct 30 05:55:15.047818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.047773 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:15.048343 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.048315 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:15.048528 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.048507 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:15.334818 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.334006 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" event={"ID":"221b4be0-d868-4358-be51-bd01846343d4","Type":"ContainerStarted","Data":"1b377b230d68a15ad7b8ecbc47e3ddd8031757fc2673bb7fe62705881cf47390"} Oct 30 05:55:15.346332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.346289 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" event={"ID":"7011a201-0f2a-490c-80fe-e9b5cea5398e","Type":"ContainerStarted","Data":"6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325"} Oct 30 05:55:15.392257 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.392228 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:55:15.393307 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.393279 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:15.393588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:15.393559 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:16.355659 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:16.355623 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" event={"ID":"221b4be0-d868-4358-be51-bd01846343d4","Type":"ContainerStarted","Data":"cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4"} Oct 30 05:55:16.389083 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:16.389038 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:16.389313 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:16.389088 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:16.423305 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:16.423257 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podStartSLOduration=3.423243009 podStartE2EDuration="3.423243009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:16.422659865 +0000 UTC m=+1370.530698899" watchObservedRunningTime="2026-10-30 05:55:16.423243009 +0000 UTC m=+1370.531282041" Oct 30 05:55:17.240343 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.240309 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:17.246081 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.246058 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:17.330686 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.330659 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2"] Oct 30 05:55:17.334254 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.334234 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.403132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.389695 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b202a0\"" Oct 30 05:55:17.403132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.389829 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b202a0-dockercfg-vnswb\"" Oct 30 05:55:17.403132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.390264 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:17.403132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.390311 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:17.423135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.420774 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhw6d\" (UniqueName: \"kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d\") pod \"payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.423135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.420849 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.575143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.572285 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhw6d\" (UniqueName: \"kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d\") pod \"payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.575143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.572353 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.575143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.573714 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.723553 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.722093 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2"] Oct 30 05:55:17.739836 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.738782 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhw6d\" (UniqueName: \"kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d\") pod \"payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.740432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.740403 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:17.759408 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.752396 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:17.759696 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:17.759677 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:55:18.025009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.001533 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:55:18.233497 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.233468 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2"] Oct 30 05:55:18.395533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.395486 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:18.395728 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.395562 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:18.628047 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.628006 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" event={"ID":"15aeb20d-5a19-4852-97b8-d0750ec71965","Type":"ContainerStarted","Data":"f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51"} Oct 30 05:55:18.628537 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.628070 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" event={"ID":"15aeb20d-5a19-4852-97b8-d0750ec71965","Type":"ContainerStarted","Data":"a2498ff56d0642cfb43ec144ac5276913d85fa0ade3c2d4b30bbf25d0f0b980d"} Oct 30 05:55:18.692612 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:18.692547 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" podStartSLOduration=1.692525364 podStartE2EDuration="1.692525364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:18.688888519 +0000 UTC m=+1372.796927556" watchObservedRunningTime="2026-10-30 05:55:18.692525364 +0000 UTC m=+1372.800564400" Oct 30 05:55:19.401128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:19.400565 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:19.401128 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:19.400624 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:19.639854 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:19.639826 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:20.393133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:20.392578 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:20.393133 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:20.392640 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:21.388910 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:21.388870 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 05:55:21.389341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:21.388939 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 05:55:22.401312 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:22.401282 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:55:22.402560 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:22.402539 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 05:55:29.713123 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.713077 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr"] Oct 30 05:55:29.718821 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.718794 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:29.750595 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.750550 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-862dc2\"" Oct 30 05:55:29.750798 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.750762 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-862dc2-dockercfg-664bk\"" Oct 30 05:55:29.819132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.805557 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr"] Oct 30 05:55:29.932129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.929623 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:29.932129 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:29.929694 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2h4r\" (UniqueName: \"kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r\") pod \"payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:30.099142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.097365 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:30.099142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.097478 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2h4r\" (UniqueName: \"kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r\") pod \"payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:30.111140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.099441 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:30.138920 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.138860 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2h4r\" (UniqueName: \"kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r\") pod \"payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:30.335671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.335642 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:30.652517 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.651571 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:55:30.665493 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.665468 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr"] Oct 30 05:55:30.667900 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:55:30.667875 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998663cb_bb11_4252_b354_ac825b8a096f.slice/crio-e20c878079e18f43d1bdc0a68c792ee78f5d33f8bf16188c644d8cc4c4b23572 WatchSource:0}: Error finding container e20c878079e18f43d1bdc0a68c792ee78f5d33f8bf16188c644d8cc4c4b23572: Status 404 returned error can't find the container with id e20c878079e18f43d1bdc0a68c792ee78f5d33f8bf16188c644d8cc4c4b23572 Oct 30 05:55:30.740871 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:30.740831 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" event={"ID":"998663cb-bb11-4252-b354-ac825b8a096f","Type":"ContainerStarted","Data":"e20c878079e18f43d1bdc0a68c792ee78f5d33f8bf16188c644d8cc4c4b23572"} Oct 30 05:55:31.749739 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:31.749705 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" event={"ID":"998663cb-bb11-4252-b354-ac825b8a096f","Type":"ContainerStarted","Data":"6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1"} Oct 30 05:55:31.750167 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:31.749794 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:31.847131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:31.841168 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" podStartSLOduration=2.841147837 podStartE2EDuration="2.841147837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:31.802580214 +0000 UTC m=+1385.910619247" watchObservedRunningTime="2026-10-30 05:55:31.841147837 +0000 UTC m=+1385.949186894" Oct 30 05:55:42.760595 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:42.760567 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 05:55:43.115957 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:55:43.115897 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:55:50.940259 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:50.940216 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:50.940754 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:50.940295 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:53.025460 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:53.025424 5980 generic.go:356] "Generic (PLEG): container finished" podID="8f345522-495a-42e2-9db8-d07dc01c289a" containerID="d197ff44b82e29408bc3a65fb6cc1b8193028bb66fc9efac8316223fe0bff036" exitCode=0 Oct 30 05:55:53.025974 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:53.025502 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" event={"ID":"8f345522-495a-42e2-9db8-d07dc01c289a","Type":"ContainerDied","Data":"d197ff44b82e29408bc3a65fb6cc1b8193028bb66fc9efac8316223fe0bff036"} Oct 30 05:55:53.026085 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:53.026048 5980 scope.go:122] "RemoveContainer" containerID="d197ff44b82e29408bc3a65fb6cc1b8193028bb66fc9efac8316223fe0bff036" Oct 30 05:55:53.028006 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:53.027974 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:53.028150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:53.028046 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:53.028150 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:53.028083 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:54.034591 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:54.034559 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" event={"ID":"8f345522-495a-42e2-9db8-d07dc01c289a","Type":"ContainerStarted","Data":"6590e91c78224afdf701fc5ff68e770926a08f624acf3e52d6264e4b7f2d89b2"} Oct 30 05:55:54.178132 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:54.175503 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:55:54.196140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:54.184256 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:55:54.196140 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:54.184333 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:55:55.176134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:55.175457 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:55:55.176134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:55.175511 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:55:56.191135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:56.185581 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:55:56.191135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:56.185667 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:55:57.178487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:57.178444 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:55:57.178700 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:57.178510 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:55:58.178135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:58.174448 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:55:58.178135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:58.174510 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:55:59.176483 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:59.176447 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 05:55:59.176759 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:55:59.176730 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 05:56:00.177999 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:00.177967 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:56:00.178477 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:00.178457 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:56:00.179085 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:00.179065 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 05:56:03.012811 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:03.012782 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm"] Oct 30 05:56:03.029996 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:03.029961 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh"] Oct 30 05:56:08.013519 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:08.013471 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerName="payload-processing" containerID="cri-o://14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8" gracePeriod=25 Oct 30 05:56:08.031190 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:08.031158 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerName="payload-pre-processing" containerID="cri-o://d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d" gracePeriod=25 Oct 30 05:56:10.768750 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:10.768713 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:11.822384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:11.822350 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:13.089799 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:56:13.089763 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:56:20.768431 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:20.768392 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:21.907610 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:21.907569 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:28.355805 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.355774 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:56:28.417162 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.417129 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz"] Oct 30 05:56:28.417999 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.417974 5980 cpu_manager.go:414] "containerMap: removing container" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerName="payload-processing" Oct 30 05:56:28.417999 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.417998 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerName="payload-processing" Oct 30 05:56:28.418318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.418300 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerName="payload-processing" Oct 30 05:56:28.423169 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.423146 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.424499 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.424472 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:56:28.447379 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.447354 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz"] Oct 30 05:56:28.530942 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.530900 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume\") pod \"b56d04d5-c6d9-496a-adf2-9417b87f800b\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " Oct 30 05:56:28.535657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.531205 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume\") pod \"45bd9927-e42c-4a6c-8940-7c090db30a62\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " Oct 30 05:56:28.535657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.531384 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd\" (UniqueName: \"kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd\") pod \"b56d04d5-c6d9-496a-adf2-9417b87f800b\" (UID: \"b56d04d5-c6d9-496a-adf2-9417b87f800b\") " Oct 30 05:56:28.535657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.531470 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn\" (UniqueName: \"kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn\") pod \"45bd9927-e42c-4a6c-8940-7c090db30a62\" (UID: \"45bd9927-e42c-4a6c-8940-7c090db30a62\") " Oct 30 05:56:28.535657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.531874 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume" pod "b56d04d5-c6d9-496a-adf2-9417b87f800b" (UID: "b56d04d5-c6d9-496a-adf2-9417b87f800b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:28.535657 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.532345 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume" pod "45bd9927-e42c-4a6c-8940-7c090db30a62" (UID: "45bd9927-e42c-4a6c-8940-7c090db30a62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:28.539315 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.539250 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd" pod "b56d04d5-c6d9-496a-adf2-9417b87f800b" (UID: "b56d04d5-c6d9-496a-adf2-9417b87f800b"). InnerVolumeSpecName "kube-api-access-67pjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:28.540752 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.540719 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8rx\" (UniqueName: \"kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx\") pod \"payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.541018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.540992 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.541276 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.541256 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45bd9927-e42c-4a6c-8940-7c090db30a62-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.541413 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.541280 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67pjd\" (UniqueName: \"kubernetes.io/projected/b56d04d5-c6d9-496a-adf2-9417b87f800b-kube-api-access-67pjd\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.541413 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.541295 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56d04d5-c6d9-496a-adf2-9417b87f800b-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.559685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.557029 5980 generic.go:356] "Generic (PLEG): container finished" podID="45bd9927-e42c-4a6c-8940-7c090db30a62" containerID="14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8" exitCode=0 Oct 30 05:56:28.559685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.557214 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" event={"ID":"45bd9927-e42c-4a6c-8940-7c090db30a62","Type":"ContainerDied","Data":"14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8"} Oct 30 05:56:28.559685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.557274 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" event={"ID":"45bd9927-e42c-4a6c-8940-7c090db30a62","Type":"ContainerDied","Data":"bd2ab30ed54d6874d313a4a0c2845a7ec1173edbd5f5e087e777f6fefd40fca5"} Oct 30 05:56:28.559685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.557305 5980 scope.go:122] "RemoveContainer" containerID="14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8" Oct 30 05:56:28.559685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.557489 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm" Oct 30 05:56:28.564169 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.562195 5980 generic.go:356] "Generic (PLEG): container finished" podID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerID="d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d" exitCode=0 Oct 30 05:56:28.564169 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.562370 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" event={"ID":"b56d04d5-c6d9-496a-adf2-9417b87f800b","Type":"ContainerDied","Data":"d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d"} Oct 30 05:56:28.564169 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.562422 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" event={"ID":"b56d04d5-c6d9-496a-adf2-9417b87f800b","Type":"ContainerDied","Data":"5f30d9246c4db5983145d1fc8b5059eb127a36a791d706eb0497e50690d4efa0"} Oct 30 05:56:28.564169 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.562503 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh" Oct 30 05:56:28.618032 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.617780 5980 scope.go:122] "RemoveContainer" containerID="14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8" Oct 30 05:56:28.618695 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:56:28.618527 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8\": container with ID starting with 14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8 not found: ID does not exist" containerID="14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8" Oct 30 05:56:28.618695 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.618567 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8"} err="failed to get container status \"14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8\": rpc error: code = NotFound desc = could not find container \"14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8\": container with ID starting with 14968f13f07d71180966e84425eb01ec62593b1e0ada83ab4a7f9a6cd5b119d8 not found: ID does not exist" Oct 30 05:56:28.618695 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.618593 5980 scope.go:122] "RemoveContainer" containerID="d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d" Oct 30 05:56:28.623376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.623348 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn" pod "45bd9927-e42c-4a6c-8940-7c090db30a62" (UID: "45bd9927-e42c-4a6c-8940-7c090db30a62"). InnerVolumeSpecName "kube-api-access-r6krn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:28.649063 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.649031 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8rx\" (UniqueName: \"kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx\") pod \"payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.651330 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.651301 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.651592 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.651578 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6krn\" (UniqueName: \"kubernetes.io/projected/45bd9927-e42c-4a6c-8940-7c090db30a62-kube-api-access-r6krn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.652433 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.652415 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.654910 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.654878 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh"] Oct 30 05:56:28.655323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.655294 5980 scope.go:122] "RemoveContainer" containerID="d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d" Oct 30 05:56:28.655708 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:56:28.655680 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d\": container with ID starting with d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d not found: ID does not exist" containerID="d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d" Oct 30 05:56:28.655853 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.655827 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d"} err="failed to get container status \"d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d\": rpc error: code = NotFound desc = could not find container \"d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d\": container with ID starting with d92839b9eb4dc6edff46df9e99a317923483fb057b5e50841fa894650ed9e14d not found: ID does not exist" Oct 30 05:56:28.657580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.657557 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-hc9fh"] Oct 30 05:56:28.680876 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.674126 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8rx\" (UniqueName: \"kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx\") pod \"payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.744609 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.744578 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:28.917221 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.916373 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm"] Oct 30 05:56:28.921238 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:28.921175 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-bstfm"] Oct 30 05:56:29.086521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.086474 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz"] Oct 30 05:56:29.092034 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:56:29.091978 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d0fa5d_2434_476d_94dc_c209b4d180f2.slice/crio-2860670e9c21960b50f17977d019f35c52db25bb9f997555765753127d391e79 WatchSource:0}: Error finding container 2860670e9c21960b50f17977d019f35c52db25bb9f997555765753127d391e79: Status 404 returned error can't find the container with id 2860670e9c21960b50f17977d019f35c52db25bb9f997555765753127d391e79 Oct 30 05:56:29.243274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.243246 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:56:29.243432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.243294 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:29.577327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.577246 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" event={"ID":"a2d0fa5d-2434-476d-94dc-c209b4d180f2","Type":"ContainerStarted","Data":"b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1"} Oct 30 05:56:29.577327 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.577308 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" event={"ID":"a2d0fa5d-2434-476d-94dc-c209b4d180f2","Type":"ContainerStarted","Data":"2860670e9c21960b50f17977d019f35c52db25bb9f997555765753127d391e79"} Oct 30 05:56:29.577869 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.577337 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:29.624738 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:29.624679 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" podStartSLOduration=60.624660012 podStartE2EDuration="1m0.624660012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:29.621930297 +0000 UTC m=+1443.729969327" watchObservedRunningTime="2026-10-30 05:56:29.624660012 +0000 UTC m=+1443.732699046" Oct 30 05:56:30.016926 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:30.016842 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bd9927-e42c-4a6c-8940-7c090db30a62" path="/var/lib/kubelet/pods/45bd9927-e42c-4a6c-8940-7c090db30a62/volumes" Oct 30 05:56:30.017384 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:30.017367 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" path="/var/lib/kubelet/pods/b56d04d5-c6d9-496a-adf2-9417b87f800b/volumes" Oct 30 05:56:32.625670 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:32.625641 5980 generic.go:356] "Generic (PLEG): container finished" podID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerID="4b465dca87930121b5f50ff395be4948e92e274ef491db4666f843ffab253002" exitCode=0 Oct 30 05:56:32.626075 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:32.625711 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" event={"ID":"1aebf5b9-d569-40b4-8dad-672fbf43ff17","Type":"ContainerDied","Data":"4b465dca87930121b5f50ff395be4948e92e274ef491db4666f843ffab253002"} Oct 30 05:56:32.626202 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:32.626186 5980 scope.go:122] "RemoveContainer" containerID="4b465dca87930121b5f50ff395be4948e92e274ef491db4666f843ffab253002" Oct 30 05:56:32.628093 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:32.628064 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:32.628220 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:32.628183 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:32.628376 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:32.628219 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:33.635703 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:33.635670 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" event={"ID":"1aebf5b9-d569-40b4-8dad-672fbf43ff17","Type":"ContainerStarted","Data":"01309b37316fdf38f2cd2f0a04a720cad0037ccaa268af2ec3e23bad618823db"} Oct 30 05:56:33.750961 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:33.750934 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:56:33.751175 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:33.751126 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:56:33.751266 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:33.751189 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 05:56:34.751443 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:34.751409 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:56:34.751887 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:34.751461 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 05:56:35.751382 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:35.751300 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:56:35.751382 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:35.751368 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 05:56:36.750614 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:36.750583 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn"] Oct 30 05:56:36.750799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:36.750778 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" containerID="cri-o://72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71" gracePeriod=30 Oct 30 05:56:36.751536 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:36.751512 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:56:36.751841 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:36.751586 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 05:56:37.752627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:37.752588 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:56:37.753181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:37.752664 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 05:56:38.751902 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:38.751868 5980 patch_prober.go:29] interesting pod/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:56:38.752071 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:38.751925 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 05:56:39.756899 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:39.756867 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:56:39.757406 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:39.757378 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:56:39.758182 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:39.758153 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 05:56:40.609193 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:40.609164 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 05:56:42.081044 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.081020 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:56:42.170256 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.170222 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp"] Oct 30 05:56:42.171472 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.171445 5980 cpu_manager.go:414] "containerMap: removing container" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerName="payload-pre-processing" Oct 30 05:56:42.171687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.171475 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerName="payload-pre-processing" Oct 30 05:56:42.171687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.171491 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" Oct 30 05:56:42.171687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.171499 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" Oct 30 05:56:42.171936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.171853 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56d04d5-c6d9-496a-adf2-9417b87f800b" containerName="payload-pre-processing" Oct 30 05:56:42.171936 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.171874 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e514947-f603-4419-a683-1f36f2ff618b" containerName="istio-proxy" Oct 30 05:56:42.175367 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.175344 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.192021 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.191996 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp"] Oct 30 05:56:42.194357 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.194331 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4fc75f-openshift-default-dockercfg-6gtb9\"" Oct 30 05:56:42.206137 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206099 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206275 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206156 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206275 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206208 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206379 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206320 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206377 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206427 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206412 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206443 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206492 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206595 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert\") pod \"3e514947-f603-4419-a683-1f36f2ff618b\" (UID: \"3e514947-f603-4419-a683-1f36f2ff618b\") " Oct 30 05:56:42.206973 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.206947 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:42.207488 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.207468 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.207630 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.207560 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:42.207722 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.207488 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:42.207964 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.207916 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:42.208346 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.208327 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:42.210623 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.210572 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:42.211768 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.211745 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:42.233272 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.233238 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "kube-api-access-kswxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:42.239192 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.239160 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy" pod "3e514947-f603-4419-a683-1f36f2ff618b" (UID: "3e514947-f603-4419-a683-1f36f2ff618b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:42.309760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.309668 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.309934 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.309842 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4zf\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310011 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.309942 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310089 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310071 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310209 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310191 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310274 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310254 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310294 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310433 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310406 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310508 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.310674 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310650 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e514947-f603-4419-a683-1f36f2ff618b-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310674 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310676 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310690 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310704 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310717 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310730 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kswxs\" (UniqueName: \"kubernetes.io/projected/3e514947-f603-4419-a683-1f36f2ff618b-kube-api-access-kswxs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310745 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e514947-f603-4419-a683-1f36f2ff618b-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.310860 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.310819 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e514947-f603-4419-a683-1f36f2ff618b-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:56:42.411831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.411799 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.411928 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4zf\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412043 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412037 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412212 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412131 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412212 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412193 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412228 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412280 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412418 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412375 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412514 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412476 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412588 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412551 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412669 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412637 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412729 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412650 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412800 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412730 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.412930 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.412911 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.414793 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.414775 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.415083 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.415064 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.437647 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.437615 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.437776 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.437659 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4zf\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.498432 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.498396 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:42.742591 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.742558 5980 generic.go:356] "Generic (PLEG): container finished" podID="3e514947-f603-4419-a683-1f36f2ff618b" containerID="72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71" exitCode=0 Oct 30 05:56:42.742775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.742620 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" event={"ID":"3e514947-f603-4419-a683-1f36f2ff618b","Type":"ContainerDied","Data":"72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71"} Oct 30 05:56:42.742775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.742624 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" Oct 30 05:56:42.742775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.742685 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn" event={"ID":"3e514947-f603-4419-a683-1f36f2ff618b","Type":"ContainerDied","Data":"a84cf059d9b1d794d2e6883c8fbc25649312217976df85fc3e3c0b03e09eeab6"} Oct 30 05:56:42.742775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.742706 5980 scope.go:122] "RemoveContainer" containerID="72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71" Oct 30 05:56:42.753728 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.753712 5980 scope.go:122] "RemoveContainer" containerID="72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71" Oct 30 05:56:42.754021 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:56:42.753998 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71\": container with ID starting with 72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71 not found: ID does not exist" containerID="72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71" Oct 30 05:56:42.754143 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.754028 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71"} err="failed to get container status \"72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71\": rpc error: code = NotFound desc = could not find container \"72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71\": container with ID starting with 72a816f0031f1889747ebc4addb831e659d82750aa9e17aafc2af4dd05806e71 not found: ID does not exist" Oct 30 05:56:42.802746 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.802719 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn"] Oct 30 05:56:42.816708 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.816646 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-g59hn"] Oct 30 05:56:42.826917 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:56:42.826886 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543da991_2779_488f_85d6_6ceeebe424fa.slice/crio-ac1c18256e54c990e86df36a1de928167f327a57b99d6ec9a83969dfac548e55 WatchSource:0}: Error finding container ac1c18256e54c990e86df36a1de928167f327a57b99d6ec9a83969dfac548e55: Status 404 returned error can't find the container with id ac1c18256e54c990e86df36a1de928167f327a57b99d6ec9a83969dfac548e55 Oct 30 05:56:42.829050 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.829021 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp"] Oct 30 05:56:42.835598 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.835563 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:42.835699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.835634 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:42.835699 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:42.835663 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:43.096592 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:56:43.096551 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:56:43.753301 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:43.753264 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" event={"ID":"543da991-2779-488f-85d6-6ceeebe424fa","Type":"ContainerStarted","Data":"6f4c7834647c50d87e0a4eee05e2f158c0fd41d2a569a6eac2a37b51e58a4d14"} Oct 30 05:56:43.753472 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:43.753325 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" event={"ID":"543da991-2779-488f-85d6-6ceeebe424fa","Type":"ContainerStarted","Data":"ac1c18256e54c990e86df36a1de928167f327a57b99d6ec9a83969dfac548e55"} Oct 30 05:56:43.839831 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:43.839778 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podStartSLOduration=9.839762806 podStartE2EDuration="9.839762806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:43.825378308 +0000 UTC m=+1457.933417342" watchObservedRunningTime="2026-10-30 05:56:43.839762806 +0000 UTC m=+1457.947801839" Oct 30 05:56:44.016288 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:44.016216 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e514947-f603-4419-a683-1f36f2ff618b" path="/var/lib/kubelet/pods/3e514947-f603-4419-a683-1f36f2ff618b/volumes" Oct 30 05:56:44.500006 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:44.499969 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:44.501965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:44.501937 5980 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:56:44.502081 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:44.501998 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:56:45.500481 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:45.500444 5980 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:56:45.500966 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:45.500512 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:56:46.499237 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:46.499199 5980 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:56:46.499466 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:46.499259 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:56:47.500139 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:47.499564 5980 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:56:47.500139 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:47.499645 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:56:48.499670 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:48.499639 5980 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:56:48.499922 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:48.499891 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:56:49.518377 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:49.518341 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:49.519666 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:49.519643 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:56:49.520300 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:56:49.520274 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 05:57:12.155561 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:12.155528 5980 generic.go:356] "Generic (PLEG): container finished" podID="221b4be0-d868-4358-be51-bd01846343d4" containerID="cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4" exitCode=0 Oct 30 05:57:12.155991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:12.155601 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" event={"ID":"221b4be0-d868-4358-be51-bd01846343d4","Type":"ContainerDied","Data":"cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4"} Oct 30 05:57:12.156077 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:12.156062 5980 scope.go:122] "RemoveContainer" containerID="cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4" Oct 30 05:57:12.157957 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:12.157929 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:12.158035 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:12.158001 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:12.158072 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:12.158037 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:13.223838 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:57:13.223794 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:57:13.244516 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:13.244473 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" event={"ID":"221b4be0-d868-4358-be51-bd01846343d4","Type":"ContainerStarted","Data":"1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710"} Oct 30 05:57:13.255207 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:13.255173 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:57:13.255402 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:13.255307 5980 patch_prober.go:29] interesting pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:57:13.255402 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:13.255368 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:57:14.240885 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:14.240858 5980 patch_prober.go:29] interesting pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:57:14.241759 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:14.241678 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:57:15.240337 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:15.240307 5980 patch_prober.go:29] interesting pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:57:15.240501 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:15.240361 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:57:16.240098 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:16.240069 5980 patch_prober.go:29] interesting pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:57:16.240508 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:16.240150 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:57:17.244287 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:17.243412 5980 patch_prober.go:29] interesting pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:57:17.244287 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:17.243498 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:57:17.752498 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:17.752464 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:57:18.240458 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:18.240428 5980 patch_prober.go:29] interesting pod/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:57:18.240627 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:18.240499 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:57:19.306967 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:19.306934 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:57:19.309521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:19.309498 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:57:24.501630 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.501597 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j"] Oct 30 05:57:24.505086 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.505067 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:24.528685 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.528653 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j"] Oct 30 05:57:24.560342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.558409 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-de7242\"" Oct 30 05:57:24.560342 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.558553 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-de7242-dockercfg-xptkd\"" Oct 30 05:57:24.592613 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.592567 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbn8\" (UniqueName: \"kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8\") pod \"payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:24.592937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.592911 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:24.812589 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.812471 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbn8\" (UniqueName: \"kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8\") pod \"payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:24.813339 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.813311 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:24.867660 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.867623 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:24.905423 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:24.875005 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbn8\" (UniqueName: \"kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8\") pod \"payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:25.121018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:25.120979 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:25.446026 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:25.445996 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j"] Oct 30 05:57:25.446846 ip-10-0-138-15 kubenswrapper[5980]: W1030 05:57:25.446818 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a73015_84a1_475b_9376_d5a149f94ae5.slice/crio-d6b502d373938818cad57e66dee6411f8ea1eeaf5bc1c93e063c8bfb6f3421a2 WatchSource:0}: Error finding container d6b502d373938818cad57e66dee6411f8ea1eeaf5bc1c93e063c8bfb6f3421a2: Status 404 returned error can't find the container with id d6b502d373938818cad57e66dee6411f8ea1eeaf5bc1c93e063c8bfb6f3421a2 Oct 30 05:57:26.381914 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:26.381871 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" event={"ID":"89a73015-84a1-475b-9376-d5a149f94ae5","Type":"ContainerStarted","Data":"85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d"} Oct 30 05:57:26.382366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:26.381946 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" event={"ID":"89a73015-84a1-475b-9376-d5a149f94ae5","Type":"ContainerStarted","Data":"d6b502d373938818cad57e66dee6411f8ea1eeaf5bc1c93e063c8bfb6f3421a2"} Oct 30 05:57:26.382366 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:26.381982 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:26.423070 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:26.423004 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" podStartSLOduration=2.422986782 podStartE2EDuration="2.422986782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:26.422182956 +0000 UTC m=+1500.530221990" watchObservedRunningTime="2026-10-30 05:57:26.422986782 +0000 UTC m=+1500.531025892" Oct 30 05:57:37.394985 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:37.394956 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 05:57:43.229137 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:57:43.228327 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:57:51.069335 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:51.069288 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:57:51.069962 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:51.069355 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:57:54.832580 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:54.832502 5980 generic.go:356] "Generic (PLEG): container finished" podID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerID="8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979" exitCode=0 Oct 30 05:57:54.833066 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:54.832576 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" event={"ID":"96352bb0-9b5b-41d8-b5cb-8877cb92fe80","Type":"ContainerDied","Data":"8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979"} Oct 30 05:57:54.833175 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:54.833078 5980 scope.go:122] "RemoveContainer" containerID="8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979" Oct 30 05:57:54.835260 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:54.835227 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:54.835377 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:54.835304 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:54.835377 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:54.835338 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:55.841593 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:55.841558 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" event={"ID":"96352bb0-9b5b-41d8-b5cb-8877cb92fe80","Type":"ContainerStarted","Data":"52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547"} Oct 30 05:57:55.927881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:55.927844 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:57:55.928073 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:55.928050 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:57:55.928163 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:55.928102 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:57:56.927530 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:56.927497 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:57:56.928021 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:56.927564 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:57:57.927671 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:57.927632 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:57:57.928185 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:57.927692 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:57:58.936130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:58.932505 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:57:58.936130 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:58.932565 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:57:59.948134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:59.927590 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:57:59.948923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:57:59.948884 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:58:00.933611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:00.933569 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:58:00.933959 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:00.933913 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:58:01.933680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:01.933632 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:58:01.935019 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:01.935000 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:58:01.935713 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:01.935697 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:58:07.129463 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:07.129031 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f"] Oct 30 05:58:07.130423 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:07.130367 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" containerID="cri-o://975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045" gracePeriod=30 Oct 30 05:58:08.033797 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:08.033758 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2"] Oct 30 05:58:08.034020 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:08.033979 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" containerID="cri-o://e44e15d66a0e302e0b215c2d4b52194d3c68565f099671d13a0fe09b599bde6d" gracePeriod=30 Oct 30 05:58:12.325135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.324266 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-dvbqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:58:12.325135 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.324333 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:58:12.857973 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.857901 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:58:12.970175 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.970099 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:12.970536 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.970511 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.970720 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.970706 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.971312 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.971033 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:12.971577 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.971562 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.972393 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.971844 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:12.972720 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.972690 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:12.972849 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.972833 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.972963 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.972950 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.973399 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.973382 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.973543 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.973527 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.974272 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.974254 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.974770 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.974751 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert\") pod \"35ab37b5-d172-4357-82c6-1e17249a4d7f\" (UID: \"35ab37b5-d172-4357-82c6-1e17249a4d7f\") " Oct 30 05:58:12.975372 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.975259 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:12.975768 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.975748 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:12.976485 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.976463 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ab37b5-d172-4357-82c6-1e17249a4d7f-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:12.976614 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.976603 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:12.976706 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.976695 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:12.976783 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.976774 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:12.976880 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.976866 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:12.976959 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.976948 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:12.978217 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.978176 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "kube-api-access-jspf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:12.980563 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.980542 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:12.981051 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:12.980981 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo" pod "35ab37b5-d172-4357-82c6-1e17249a4d7f" (UID: "35ab37b5-d172-4357-82c6-1e17249a4d7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:13.078937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.078906 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.078937 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.078936 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ab37b5-d172-4357-82c6-1e17249a4d7f-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.079187 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.078951 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspf6\" (UniqueName: \"kubernetes.io/projected/35ab37b5-d172-4357-82c6-1e17249a4d7f-kube-api-access-jspf6\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.091169 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:58:13.091136 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.255866 5980 generic.go:356] "Generic (PLEG): container finished" podID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerID="e44e15d66a0e302e0b215c2d4b52194d3c68565f099671d13a0fe09b599bde6d" exitCode=0 Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.256151 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" event={"ID":"3c3b2678-6a74-406f-89c9-e3247e5fc2b9","Type":"ContainerDied","Data":"e44e15d66a0e302e0b215c2d4b52194d3c68565f099671d13a0fe09b599bde6d"} Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.257815 5980 generic.go:356] "Generic (PLEG): container finished" podID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerID="975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045" exitCode=0 Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.257869 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" event={"ID":"35ab37b5-d172-4357-82c6-1e17249a4d7f","Type":"ContainerDied","Data":"975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045"} Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.257908 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" event={"ID":"35ab37b5-d172-4357-82c6-1e17249a4d7f","Type":"ContainerDied","Data":"396473051e7cc7086ef4d34a449b3f3c669bffb1588ae620092821d149c30401"} Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.257936 5980 scope.go:122] "RemoveContainer" containerID="975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045" Oct 30 05:58:13.261195 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.258133 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f" Oct 30 05:58:13.276597 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.276570 5980 scope.go:122] "RemoveContainer" containerID="975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045" Oct 30 05:58:13.276991 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:58:13.276960 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045\": container with ID starting with 975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045 not found: ID does not exist" containerID="975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045" Oct 30 05:58:13.277089 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.277001 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045"} err="failed to get container status \"975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045\": rpc error: code = NotFound desc = could not find container \"975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045\": container with ID starting with 975621effb3c429ff77b1574e4fdab3db1c094a01c9d35bbafe91dc44165f045 not found: ID does not exist" Oct 30 05:58:13.324504 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.324460 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f"] Oct 30 05:58:13.335290 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.335263 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-b7m5f"] Oct 30 05:58:13.370184 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.370157 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:58:13.395992 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.395957 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396057 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396154 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396323 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396307 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396417 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396401 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396482 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396405 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.396482 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396470 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:13.396582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396487 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396582 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396560 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396680 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396641 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.396735 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396684 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.396958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396739 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.396958 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.396767 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs\") pod \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\" (UID: \"3c3b2678-6a74-406f-89c9-e3247e5fc2b9\") " Oct 30 05:58:13.397090 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.397047 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.397883 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.397859 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.398005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.397887 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.398005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.397901 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.398005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.397915 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.398005 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.397929 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.399152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.399101 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:13.399923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.399328 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.399923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.399665 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.399923 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.399790 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm" pod "3c3b2678-6a74-406f-89c9-e3247e5fc2b9" (UID: "3c3b2678-6a74-406f-89c9-e3247e5fc2b9"). InnerVolumeSpecName "kube-api-access-mfkfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.500052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.500015 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.500052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.500044 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.500052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.500053 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfkfm\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-kube-api-access-mfkfm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.500337 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:13.500062 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c3b2678-6a74-406f-89c9-e3247e5fc2b9-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:58:14.016053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:14.016021 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" path="/var/lib/kubelet/pods/35ab37b5-d172-4357-82c6-1e17249a4d7f/volumes" Oct 30 05:58:14.288259 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:14.288175 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" event={"ID":"3c3b2678-6a74-406f-89c9-e3247e5fc2b9","Type":"ContainerDied","Data":"5f73ac2c80846692855344f563ff3d782513b75d25544f52ca325ade575c5527"} Oct 30 05:58:14.288466 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:14.288454 5980 scope.go:122] "RemoveContainer" containerID="e44e15d66a0e302e0b215c2d4b52194d3c68565f099671d13a0fe09b599bde6d" Oct 30 05:58:14.288788 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:14.288703 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2" Oct 30 05:58:14.355611 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:14.355580 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2"] Oct 30 05:58:14.370181 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:14.370092 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7ncm2"] Oct 30 05:58:16.017522 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:16.017488 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" path="/var/lib/kubelet/pods/3c3b2678-6a74-406f-89c9-e3247e5fc2b9/volumes" Oct 30 05:58:43.308418 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:58:43.308378 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:58:56.906877 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:56.906840 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v"] Oct 30 05:58:56.907791 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:56.907738 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" containerID="cri-o://52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547" gracePeriod=30 Oct 30 05:58:57.152134 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:57.144154 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb"] Oct 30 05:58:57.152706 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:58:57.152641 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" containerID="cri-o://1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710" gracePeriod=30 Oct 30 05:59:01.842429 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:01.842393 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx"] Oct 30 05:59:01.842908 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:01.842637 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" containerID="cri-o://98b814a27c6c77b61ff2007927c2fc5b049b2b84d76288f983cf56a6c3b80f4f" gracePeriod=30 Oct 30 05:59:01.855938 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:01.855893 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7"] Oct 30 05:59:01.856233 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:01.856197 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" containerID="cri-o://1aa064af6d4f2d7eff2d983f25b10449bdfeac613596d8c326018765f9201c2f" gracePeriod=30 Oct 30 05:59:01.937147 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:01.937092 5980 patch_prober.go:29] interesting pod/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:59:01.937264 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:01.937180 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 05:59:02.291302 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.291267 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:59:02.370794 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.370701 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.370976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.370794 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.370976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.370849 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.370976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.370890 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.370976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.370955 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.371289 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.371020 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.371358 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.371334 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.371421 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.371399 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.371490 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.371433 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo\") pod \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\" (UID: \"96352bb0-9b5b-41d8-b5cb-8877cb92fe80\") " Oct 30 05:59:02.373694 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.373658 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.373899 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.373878 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.374158 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.374089 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.374471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.374434 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:02.375159 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.375133 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.381775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.381722 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:02.382788 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.382753 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "kube-api-access-qrr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:02.392446 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.392408 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.393295 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.393254 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo" pod "96352bb0-9b5b-41d8-b5cb-8877cb92fe80" (UID: "96352bb0-9b5b-41d8-b5cb-8877cb92fe80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:02.439775 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.439320 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:59:02.474045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474006 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474038 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474045 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474050 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474062 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrr5v\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-kube-api-access-qrr5v\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474074 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474085 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474095 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474122 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.474332 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.474134 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96352bb0-9b5b-41d8-b5cb-8877cb92fe80-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.575318 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575280 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575338 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575427 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575503 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575705 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575667 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575811 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575705 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575811 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575731 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.575811 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575805 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.576051 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.575911 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token\") pod \"221b4be0-d868-4358-be51-bd01846343d4\" (UID: \"221b4be0-d868-4358-be51-bd01846343d4\") " Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.576189 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.576823 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.577284 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.578074 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.578381 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.578642 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.582297 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.585276 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.587282 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:02.589906 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.587294 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t" pod "221b4be0-d868-4358-be51-bd01846343d4" (UID: "221b4be0-d868-4358-be51-bd01846343d4"). InnerVolumeSpecName "kube-api-access-7pd7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:02.678565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678471 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b4be0-d868-4358-be51-bd01846343d4-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678509 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b4be0-d868-4358-be51-bd01846343d4-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678522 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678535 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678550 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678562 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678575 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b4be0-d868-4358-be51-bd01846343d4-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.678881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.678588 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pd7t\" (UniqueName: \"kubernetes.io/projected/221b4be0-d868-4358-be51-bd01846343d4-kube-api-access-7pd7t\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:02.976311 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.976275 5980 generic.go:356] "Generic (PLEG): container finished" podID="221b4be0-d868-4358-be51-bd01846343d4" containerID="1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710" exitCode=0 Oct 30 05:59:02.976799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.976328 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" Oct 30 05:59:02.976799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.976362 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" event={"ID":"221b4be0-d868-4358-be51-bd01846343d4","Type":"ContainerDied","Data":"1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710"} Oct 30 05:59:02.976799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.976420 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb" event={"ID":"221b4be0-d868-4358-be51-bd01846343d4","Type":"ContainerDied","Data":"1b377b230d68a15ad7b8ecbc47e3ddd8031757fc2673bb7fe62705881cf47390"} Oct 30 05:59:02.976799 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.976445 5980 scope.go:122] "RemoveContainer" containerID="1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710" Oct 30 05:59:02.978897 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.978872 5980 generic.go:356] "Generic (PLEG): container finished" podID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerID="52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547" exitCode=0 Oct 30 05:59:02.979009 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.978939 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" Oct 30 05:59:02.979057 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.978944 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" event={"ID":"96352bb0-9b5b-41d8-b5cb-8877cb92fe80","Type":"ContainerDied","Data":"52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547"} Oct 30 05:59:02.979102 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.979058 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v" event={"ID":"96352bb0-9b5b-41d8-b5cb-8877cb92fe80","Type":"ContainerDied","Data":"4900ad1be121c701d9bd7439de5dbfe3fc0035ab2b371386cd40d032d97418c1"} Oct 30 05:59:02.998142 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:02.995890 5980 scope.go:122] "RemoveContainer" containerID="cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4" Oct 30 05:59:03.007881 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.007854 5980 scope.go:122] "RemoveContainer" containerID="1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710" Oct 30 05:59:03.008245 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:03.008225 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710\": container with ID starting with 1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710 not found: ID does not exist" containerID="1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710" Oct 30 05:59:03.008341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.008258 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710"} err="failed to get container status \"1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710\": rpc error: code = NotFound desc = could not find container \"1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710\": container with ID starting with 1b283333be713ebfc3f4635a11523c1b01b695598eea6852d8c239c5b6770710 not found: ID does not exist" Oct 30 05:59:03.008341 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.008281 5980 scope.go:122] "RemoveContainer" containerID="cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4" Oct 30 05:59:03.008615 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:03.008593 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4\": container with ID starting with cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4 not found: ID does not exist" containerID="cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4" Oct 30 05:59:03.008664 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.008626 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4"} err="failed to get container status \"cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4\": rpc error: code = NotFound desc = could not find container \"cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4\": container with ID starting with cf7f0c0794e8b22ce369eb5cbb6bc2954b400c00b961642bb65a95a96295eca4 not found: ID does not exist" Oct 30 05:59:03.008664 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.008648 5980 scope.go:122] "RemoveContainer" containerID="52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547" Oct 30 05:59:03.019526 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.019505 5980 scope.go:122] "RemoveContainer" containerID="8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979" Oct 30 05:59:03.029336 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.029313 5980 scope.go:122] "RemoveContainer" containerID="52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547" Oct 30 05:59:03.029647 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:03.029624 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547\": container with ID starting with 52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547 not found: ID does not exist" containerID="52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547" Oct 30 05:59:03.029715 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.029656 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547"} err="failed to get container status \"52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547\": rpc error: code = NotFound desc = could not find container \"52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547\": container with ID starting with 52c7b8a7a8447322acda6d52c9c1abb9f4d65856b69c9c5acd8671f42f321547 not found: ID does not exist" Oct 30 05:59:03.029715 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.029675 5980 scope.go:122] "RemoveContainer" containerID="8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979" Oct 30 05:59:03.029945 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:03.029924 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979\": container with ID starting with 8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979 not found: ID does not exist" containerID="8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979" Oct 30 05:59:03.029991 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.029956 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979"} err="failed to get container status \"8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979\": rpc error: code = NotFound desc = could not find container \"8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979\": container with ID starting with 8317d4e4a7805e6eccccb60ebaa87bdfe3f78ccf68d74af10610e326789e0979 not found: ID does not exist" Oct 30 05:59:03.039029 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.038999 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v"] Oct 30 05:59:03.052377 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.052347 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-3571f8-openshift-default-5f85d888fc-lfr6v"] Oct 30 05:59:03.097687 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.097657 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb"] Oct 30 05:59:03.103273 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.103244 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a3efab09-openshift-default-789b59cc4f-wn9kb"] Oct 30 05:59:03.620623 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:03.620578 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2"] Oct 30 05:59:04.089136 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:04.089040 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221b4be0-d868-4358-be51-bd01846343d4" path="/var/lib/kubelet/pods/221b4be0-d868-4358-be51-bd01846343d4/volumes" Oct 30 05:59:04.096504 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:04.090894 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" path="/var/lib/kubelet/pods/96352bb0-9b5b-41d8-b5cb-8877cb92fe80/volumes" Oct 30 05:59:07.096131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.095613 5980 generic.go:356] "Generic (PLEG): container finished" podID="84130c06-dc82-437e-a335-5297669035ad" containerID="98b814a27c6c77b61ff2007927c2fc5b049b2b84d76288f983cf56a6c3b80f4f" exitCode=0 Oct 30 05:59:07.096131 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.095853 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" event={"ID":"84130c06-dc82-437e-a335-5297669035ad","Type":"ContainerDied","Data":"98b814a27c6c77b61ff2007927c2fc5b049b2b84d76288f983cf56a6c3b80f4f"} Oct 30 05:59:07.100187 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.099058 5980 generic.go:356] "Generic (PLEG): container finished" podID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerID="1aa064af6d4f2d7eff2d983f25b10449bdfeac613596d8c326018765f9201c2f" exitCode=0 Oct 30 05:59:07.100187 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.099169 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" event={"ID":"648bda6f-2cef-47f5-a7fa-5d571cfa842d","Type":"ContainerDied","Data":"1aa064af6d4f2d7eff2d983f25b10449bdfeac613596d8c326018765f9201c2f"} Oct 30 05:59:07.201561 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.201341 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269026 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269098 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269256 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269359 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269451 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269497 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269547 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269618 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.269683 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo\") pod \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\" (UID: \"648bda6f-2cef-47f5-a7fa-5d571cfa842d\") " Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.271285 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.271721 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.271889 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.273041 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.273258 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.277315 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:07.282001 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.279670 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "kube-api-access-ztzdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:07.283509 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.283468 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.286628 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.286062 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:59:07.293097 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.293056 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo" pod "648bda6f-2cef-47f5-a7fa-5d571cfa842d" (UID: "648bda6f-2cef-47f5-a7fa-5d571cfa842d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:07.377940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.377821 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.377940 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.377903 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.377969 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378010 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378087 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378166 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378201 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378198 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378516 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378235 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378516 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378425 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs\") pod \"84130c06-dc82-437e-a335-5297669035ad\" (UID: \"84130c06-dc82-437e-a335-5297669035ad\") " Oct 30 05:59:07.378714 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.378684 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.379060 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379033 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:07.379409 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379391 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.379521 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379510 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.379601 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379592 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.379684 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379675 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.379760 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379750 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.379847 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379824 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84130c06-dc82-437e-a335-5297669035ad-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.379847 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379845 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.380031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379859 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.380031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379871 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.380031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379885 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzdh\" (UniqueName: \"kubernetes.io/projected/648bda6f-2cef-47f5-a7fa-5d571cfa842d-kube-api-access-ztzdh\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.380031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379898 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648bda6f-2cef-47f5-a7fa-5d571cfa842d-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.380031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379452 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.380031 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.379228 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.380467 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.380138 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.384089 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.382493 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:07.384089 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.382514 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:07.385152 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.385079 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "kube-api-access-wbzpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:07.386559 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.386535 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token" pod "84130c06-dc82-437e-a335-5297669035ad" (UID: "84130c06-dc82-437e-a335-5297669035ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:07.481989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.481954 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.481989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.481988 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.481989 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.481999 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.482253 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.482010 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzpq\" (UniqueName: \"kubernetes.io/projected/84130c06-dc82-437e-a335-5297669035ad-kube-api-access-wbzpq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.482253 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.482023 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.482253 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.482035 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84130c06-dc82-437e-a335-5297669035ad-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.482253 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.482045 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84130c06-dc82-437e-a335-5297669035ad-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:07.918380 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:07.918343 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd"] Oct 30 05:59:08.109471 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.109437 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" Oct 30 05:59:08.110125 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.109464 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7" event={"ID":"648bda6f-2cef-47f5-a7fa-5d571cfa842d","Type":"ContainerDied","Data":"491603d1ad9f6ecf1fd3ad4c75befa49d8fa7402439131212579560c65b6085c"} Oct 30 05:59:08.110125 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.109532 5980 scope.go:122] "RemoveContainer" containerID="1aa064af6d4f2d7eff2d983f25b10449bdfeac613596d8c326018765f9201c2f" Oct 30 05:59:08.112018 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.111971 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" event={"ID":"84130c06-dc82-437e-a335-5297669035ad","Type":"ContainerDied","Data":"8d4e8a3ce289098cec19303c672a34e7c0076fbfd129313aceb855a82afdb6a5"} Oct 30 05:59:08.112168 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.111999 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx" Oct 30 05:59:08.123583 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.123558 5980 scope.go:122] "RemoveContainer" containerID="98b814a27c6c77b61ff2007927c2fc5b049b2b84d76288f983cf56a6c3b80f4f" Oct 30 05:59:08.214159 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.211600 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx"] Oct 30 05:59:08.215981 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.215943 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-q6mdx"] Oct 30 05:59:08.221720 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.221560 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd"] Oct 30 05:59:08.221888 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.221841 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" containerID="cri-o://921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c" gracePeriod=30 Oct 30 05:59:08.269206 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.269168 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7"] Oct 30 05:59:08.269403 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.269387 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-gvmf7"] Oct 30 05:59:08.626186 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:08.622070 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" podUID="15aeb20d-5a19-4852-97b8-d0750ec71965" containerName="payload-processing" containerID="cri-o://f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51" gracePeriod=25 Oct 30 05:59:09.020880 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.020855 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:59:09.095328 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.095289 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume\") pod \"15aeb20d-5a19-4852-97b8-d0750ec71965\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " Oct 30 05:59:09.095584 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.095553 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d\" (UniqueName: \"kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d\") pod \"15aeb20d-5a19-4852-97b8-d0750ec71965\" (UID: \"15aeb20d-5a19-4852-97b8-d0750ec71965\") " Oct 30 05:59:09.096029 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.096006 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume" pod "15aeb20d-5a19-4852-97b8-d0750ec71965" (UID: "15aeb20d-5a19-4852-97b8-d0750ec71965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:09.096628 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.096605 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15aeb20d-5a19-4852-97b8-d0750ec71965-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:09.099283 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.099239 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d" pod "15aeb20d-5a19-4852-97b8-d0750ec71965" (UID: "15aeb20d-5a19-4852-97b8-d0750ec71965"). InnerVolumeSpecName "kube-api-access-zhw6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:09.122970 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.122937 5980 generic.go:356] "Generic (PLEG): container finished" podID="15aeb20d-5a19-4852-97b8-d0750ec71965" containerID="f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51" exitCode=0 Oct 30 05:59:09.123459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.122986 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" Oct 30 05:59:09.123459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.123177 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" event={"ID":"15aeb20d-5a19-4852-97b8-d0750ec71965","Type":"ContainerDied","Data":"f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51"} Oct 30 05:59:09.123459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.123233 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2" event={"ID":"15aeb20d-5a19-4852-97b8-d0750ec71965","Type":"ContainerDied","Data":"a2498ff56d0642cfb43ec144ac5276913d85fa0ade3c2d4b30bbf25d0f0b980d"} Oct 30 05:59:09.123459 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.123267 5980 scope.go:122] "RemoveContainer" containerID="f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51" Oct 30 05:59:09.148523 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.148497 5980 scope.go:122] "RemoveContainer" containerID="f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51" Oct 30 05:59:09.148881 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:09.148858 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51\": container with ID starting with f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51 not found: ID does not exist" containerID="f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51" Oct 30 05:59:09.148965 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.148894 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51"} err="failed to get container status \"f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51\": rpc error: code = NotFound desc = could not find container \"f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51\": container with ID starting with f6a65326901475a653ce46046310d00981949ec0cd452c6ab420e3c5a575cf51 not found: ID does not exist" Oct 30 05:59:09.183243 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.183194 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2"] Oct 30 05:59:09.187189 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.187138 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b202a0-579d586c5f-5r2k2"] Oct 30 05:59:09.198505 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:09.198477 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhw6d\" (UniqueName: \"kubernetes.io/projected/15aeb20d-5a19-4852-97b8-d0750ec71965-kube-api-access-zhw6d\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.016976 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:10.016942 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15aeb20d-5a19-4852-97b8-d0750ec71965" path="/var/lib/kubelet/pods/15aeb20d-5a19-4852-97b8-d0750ec71965/volumes" Oct 30 05:59:10.017532 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:10.017516 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" path="/var/lib/kubelet/pods/648bda6f-2cef-47f5-a7fa-5d571cfa842d/volumes" Oct 30 05:59:10.018000 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:10.017987 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84130c06-dc82-437e-a335-5297669035ad" path="/var/lib/kubelet/pods/84130c06-dc82-437e-a335-5297669035ad/volumes" Oct 30 05:59:12.919213 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:12.919169 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" podUID="8a668c6d-8b3d-4fdd-8218-d928cc1462db" containerName="payload-processing" containerID="cri-o://bc7c8da89823ea6f390781e46b0c1e6465852bda305fbd43c98aa1c2fb5852d9" gracePeriod=25 Oct 30 05:59:13.095880 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:13.095830 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 05:59:13.336480 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.329098 5980 generic.go:356] "Generic (PLEG): container finished" podID="8a668c6d-8b3d-4fdd-8218-d928cc1462db" containerID="bc7c8da89823ea6f390781e46b0c1e6465852bda305fbd43c98aa1c2fb5852d9" exitCode=0 Oct 30 05:59:13.336480 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.329417 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" event={"ID":"8a668c6d-8b3d-4fdd-8218-d928cc1462db","Type":"ContainerDied","Data":"bc7c8da89823ea6f390781e46b0c1e6465852bda305fbd43c98aa1c2fb5852d9"} Oct 30 05:59:13.405541 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.405506 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:59:13.496191 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.496158 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl\" (UniqueName: \"kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl\") pod \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " Oct 30 05:59:13.496460 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.496446 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume\") pod \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\" (UID: \"8a668c6d-8b3d-4fdd-8218-d928cc1462db\") " Oct 30 05:59:13.497299 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.497263 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume" pod "8a668c6d-8b3d-4fdd-8218-d928cc1462db" (UID: "8a668c6d-8b3d-4fdd-8218-d928cc1462db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:13.499862 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.499840 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl" pod "8a668c6d-8b3d-4fdd-8218-d928cc1462db" (UID: "8a668c6d-8b3d-4fdd-8218-d928cc1462db"). InnerVolumeSpecName "kube-api-access-4sxwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:13.539985 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.539958 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:59:13.597635 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.597596 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.597833 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.597704 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.597833 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.597810 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.597950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.597866 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.597950 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.597895 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.598052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.597977 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.598052 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598023 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.598229 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598043 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:13.598289 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598272 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.598691 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598371 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs\") pod \"85ba01ac-dfc1-43f0-8550-d101eeddb858\" (UID: \"85ba01ac-dfc1-43f0-8550-d101eeddb858\") " Oct 30 05:59:13.598691 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598403 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:13.598691 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598612 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:13.598945 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.598814 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:13.599647 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599619 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.599761 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599650 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a668c6d-8b3d-4fdd-8218-d928cc1462db-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.599761 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599665 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.599761 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599678 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85ba01ac-dfc1-43f0-8550-d101eeddb858-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.599761 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599692 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sxwl\" (UniqueName: \"kubernetes.io/projected/8a668c6d-8b3d-4fdd-8218-d928cc1462db-kube-api-access-4sxwl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.599761 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599707 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.600053 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.599951 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:13.601766 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.601740 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:13.602091 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.602055 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:13.602288 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.602261 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:13.602574 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.602551 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k" pod "85ba01ac-dfc1-43f0-8550-d101eeddb858" (UID: "85ba01ac-dfc1-43f0-8550-d101eeddb858"). InnerVolumeSpecName "kube-api-access-krw6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:13.709510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.706992 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krw6k\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-kube-api-access-krw6k\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.709510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.707032 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.709510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.707046 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.709510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.707058 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.709510 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:13.707070 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85ba01ac-dfc1-43f0-8550-d101eeddb858-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 05:59:14.337994 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.337964 5980 generic.go:356] "Generic (PLEG): container finished" podID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerID="921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c" exitCode=0 Oct 30 05:59:14.338487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.338013 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" Oct 30 05:59:14.338487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.338057 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" event={"ID":"85ba01ac-dfc1-43f0-8550-d101eeddb858","Type":"ContainerDied","Data":"921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c"} Oct 30 05:59:14.338487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.338159 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd" event={"ID":"85ba01ac-dfc1-43f0-8550-d101eeddb858","Type":"ContainerDied","Data":"bf6a1e4e1019ee3dc2f109ae7d73e4d217972c79901240b51ef7e0cd6ab70d47"} Oct 30 05:59:14.338487 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.338196 5980 scope.go:122] "RemoveContainer" containerID="921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c" Oct 30 05:59:14.339502 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.339487 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" Oct 30 05:59:14.339565 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.339526 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd" event={"ID":"8a668c6d-8b3d-4fdd-8218-d928cc1462db","Type":"ContainerDied","Data":"79a31a7ad43f87b5bdb69a1ddc6715954dd479976a5c94e9e2e9f352ab6de09d"} Oct 30 05:59:14.348711 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.348696 5980 scope.go:122] "RemoveContainer" containerID="921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c" Oct 30 05:59:14.348941 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:14.348915 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c\": container with ID starting with 921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c not found: ID does not exist" containerID="921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c" Oct 30 05:59:14.349049 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.348947 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c"} err="failed to get container status \"921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c\": rpc error: code = NotFound desc = could not find container \"921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c\": container with ID starting with 921cd64074c922df0f173eaadc484c1a87d24ab2da07554982262288b3042f3c not found: ID does not exist" Oct 30 05:59:14.349049 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.348970 5980 scope.go:122] "RemoveContainer" containerID="bc7c8da89823ea6f390781e46b0c1e6465852bda305fbd43c98aa1c2fb5852d9" Oct 30 05:59:14.410062 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.410027 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd"] Oct 30 05:59:14.423659 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.423631 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-hlpxd"] Oct 30 05:59:14.451359 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.451327 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd"] Oct 30 05:59:14.454449 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:14.454426 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4fc75f-847c6dd8b7-dndjd"] Oct 30 05:59:16.016533 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:16.016501 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" path="/var/lib/kubelet/pods/85ba01ac-dfc1-43f0-8550-d101eeddb858/volumes" Oct 30 05:59:16.016983 ip-10-0-138-15 kubenswrapper[5980]: I1030 05:59:16.016969 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a668c6d-8b3d-4fdd-8218-d928cc1462db" path="/var/lib/kubelet/pods/8a668c6d-8b3d-4fdd-8218-d928cc1462db/volumes" Oct 30 05:59:43.089257 ip-10-0-138-15 kubenswrapper[5980]: E1030 05:59:43.089160 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:00:00.201511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.201473 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk"] Oct 30 06:00:00.203429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203398 5980 cpu_manager.go:414] "containerMap: removing container" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" Oct 30 06:00:00.203429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203427 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203462 5980 cpu_manager.go:414] "containerMap: removing container" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203470 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203487 5980 cpu_manager.go:414] "containerMap: removing container" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203494 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203518 5980 cpu_manager.go:414] "containerMap: removing container" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203525 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203537 5980 cpu_manager.go:414] "containerMap: removing container" podUID="15aeb20d-5a19-4852-97b8-d0750ec71965" containerName="payload-processing" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203544 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15aeb20d-5a19-4852-97b8-d0750ec71965" containerName="payload-processing" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203564 5980 cpu_manager.go:414] "containerMap: removing container" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203571 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203595 5980 cpu_manager.go:414] "containerMap: removing container" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203602 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203618 5980 cpu_manager.go:414] "containerMap: removing container" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" Oct 30 06:00:00.203629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203624 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203640 5980 cpu_manager.go:414] "containerMap: removing container" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203647 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203660 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203667 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203683 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8a668c6d-8b3d-4fdd-8218-d928cc1462db" containerName="payload-processing" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203689 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a668c6d-8b3d-4fdd-8218-d928cc1462db" containerName="payload-processing" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203834 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="15aeb20d-5a19-4852-97b8-d0750ec71965" containerName="payload-processing" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203848 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203858 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203869 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="648bda6f-2cef-47f5-a7fa-5d571cfa842d" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203879 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="84130c06-dc82-437e-a335-5297669035ad" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203888 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3b2678-6a74-406f-89c9-e3247e5fc2b9" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203898 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a668c6d-8b3d-4fdd-8218-d928cc1462db" containerName="payload-processing" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203908 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ab37b5-d172-4357-82c6-1e17249a4d7f" containerName="istio-proxy" Oct 30 06:00:00.205512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.203919 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="85ba01ac-dfc1-43f0-8550-d101eeddb858" containerName="istio-proxy" Oct 30 06:00:00.215389 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.214263 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.216393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.216365 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl"] Oct 30 06:00:00.217219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.217203 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="96352bb0-9b5b-41d8-b5cb-8877cb92fe80" containerName="istio-proxy" Oct 30 06:00:00.217288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.217222 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="221b4be0-d868-4358-be51-bd01846343d4" containerName="istio-proxy" Oct 30 06:00:00.227204 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.227177 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.230684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.230657 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b"] Oct 30 06:00:00.257965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.257933 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk"] Oct 30 06:00:00.258496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.258098 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.265075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.265051 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl"] Oct 30 06:00:00.272276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.272247 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b"] Oct 30 06:00:00.278271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.278246 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg"] Oct 30 06:00:00.285206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.285172 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:00:00.285803 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.285452 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.299452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.298638 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg"] Oct 30 06:00:00.338742 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.338711 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zc7t\" (UniqueName: \"kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t\") pod \"maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.338952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.338932 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889000-lkffl\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.339097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.339073 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmg4\" (UniqueName: \"kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4\") pod \"maas-api-key-cleanup-29889000-lkffl\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.339275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.339256 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.440638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.440606 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889000-lkffl\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.440841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.440649 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsb7\" (UniqueName: \"kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.440841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.440735 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmg4\" (UniqueName: \"kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4\") pod \"maas-api-key-cleanup-29889000-lkffl\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.440841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.440802 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.440841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.440828 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8zk6\" (UniqueName: \"kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.441068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.440974 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.441145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.441126 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zc7t\" (UniqueName: \"kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t\") pod \"maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.441213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.441200 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.441470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.441447 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889000-lkffl\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.441684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.441665 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.466701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.466631 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmg4\" (UniqueName: \"kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4\") pod \"maas-api-key-cleanup-29889000-lkffl\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.471269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.471238 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zc7t\" (UniqueName: \"kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t\") pod \"maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.542722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.542682 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.542918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.542729 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8zk6\" (UniqueName: \"kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.542991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.542923 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.543046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.543031 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsb7\" (UniqueName: \"kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.543518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.543492 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.543811 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.543790 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.557298 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.557273 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:00.568807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.568773 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsb7\" (UniqueName: \"kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.569137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.569093 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8zk6\" (UniqueName: \"kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.571866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.571839 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:00.593004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.592769 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:00.606648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.606418 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:00.896410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.896376 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk"] Oct 30 06:00:00.897227 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:00:00.897196 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445d826a_7d5f_442e_9b68_7aa53a386ab9.slice/crio-03f14fc25ad231b510a86079a6c0685f1f6021d6ca1417af66c2cb08a602e334 WatchSource:0}: Error finding container 03f14fc25ad231b510a86079a6c0685f1f6021d6ca1417af66c2cb08a602e334: Status 404 returned error can't find the container with id 03f14fc25ad231b510a86079a6c0685f1f6021d6ca1417af66c2cb08a602e334 Oct 30 06:00:00.899433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.899418 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:00:00.917640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.917608 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl"] Oct 30 06:00:00.973204 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:00:00.973172 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fe45ef_0047_4ffd_ae46_793506291ae4.slice/crio-403b1595398d07a266c9a6c7c46be5bb88a74b77984ec2a82e4c90663fd301b1 WatchSource:0}: Error finding container 403b1595398d07a266c9a6c7c46be5bb88a74b77984ec2a82e4c90663fd301b1: Status 404 returned error can't find the container with id 403b1595398d07a266c9a6c7c46be5bb88a74b77984ec2a82e4c90663fd301b1 Oct 30 06:00:00.973361 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.973342 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b"] Oct 30 06:00:00.991209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:00.990972 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg"] Oct 30 06:00:01.946044 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:01.946007 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerStarted","Data":"403b1595398d07a266c9a6c7c46be5bb88a74b77984ec2a82e4c90663fd301b1"} Oct 30 06:00:01.948370 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:01.948327 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerStarted","Data":"3cac0d46156eb2a39daaf2f43d94096e98c048396baad4be43cf462ce54cd989"} Oct 30 06:00:01.953794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:01.953763 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerStarted","Data":"03f14fc25ad231b510a86079a6c0685f1f6021d6ca1417af66c2cb08a602e334"} Oct 30 06:00:01.955418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:01.955394 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerStarted","Data":"b99bd986fe440996c30619e79978bc7c8e7e676da86a8d49c001bc066e753721"} Oct 30 06:00:03.119193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:03.119154 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw"] Oct 30 06:00:03.122842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:03.120164 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" containerID="cri-o://01309b37316fdf38f2cd2f0a04a720cad0037ccaa268af2ec3e23bad618823db" gracePeriod=30 Oct 30 06:00:08.440419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.440380 5980 generic.go:356] "Generic (PLEG): container finished" podID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerID="01309b37316fdf38f2cd2f0a04a720cad0037ccaa268af2ec3e23bad618823db" exitCode=0 Oct 30 06:00:08.440793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.440431 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" event={"ID":"1aebf5b9-d569-40b4-8dad-672fbf43ff17","Type":"ContainerDied","Data":"01309b37316fdf38f2cd2f0a04a720cad0037ccaa268af2ec3e23bad618823db"} Oct 30 06:00:08.440793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.440493 5980 scope.go:122] "RemoveContainer" containerID="4b465dca87930121b5f50ff395be4948e92e274ef491db4666f843ffab253002" Oct 30 06:00:08.688816 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.688587 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 06:00:08.852548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.852520 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.852699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.852568 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.852699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.852664 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.852806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.852790 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.852871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.852856 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.852999 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.852983 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.853071 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.853037 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.853138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.853123 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.853218 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.853204 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy\") pod \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\" (UID: \"1aebf5b9-d569-40b4-8dad-672fbf43ff17\") " Oct 30 06:00:08.854745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.854717 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:08.855058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.855036 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:08.855289 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.855272 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:08.857236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.857212 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:08.857686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.857666 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "kube-api-access-sj4rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:08.857975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.857957 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:08.858075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.858047 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:08.864207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.864185 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:08.864335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.864318 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo" pod "1aebf5b9-d569-40b4-8dad-672fbf43ff17" (UID: "1aebf5b9-d569-40b4-8dad-672fbf43ff17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955814 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955847 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955862 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955877 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj4rg\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-kube-api-access-sj4rg\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955893 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955908 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955921 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955933 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:08.956037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:08.955945 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aebf5b9-d569-40b4-8dad-672fbf43ff17-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:09.452650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:09.452609 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" event={"ID":"1aebf5b9-d569-40b4-8dad-672fbf43ff17","Type":"ContainerDied","Data":"4765d0a53563e2e55b76e3e56753e8e85b75528e1f708627baf1441089187303"} Oct 30 06:00:09.453039 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:09.452683 5980 scope.go:122] "RemoveContainer" containerID="01309b37316fdf38f2cd2f0a04a720cad0037ccaa268af2ec3e23bad618823db" Oct 30 06:00:09.453039 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:09.452808 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw" Oct 30 06:00:09.527366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:09.527328 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw"] Oct 30 06:00:09.541459 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:09.541424 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b202a0-openshift-default-5756cfdf4f-fg7jw"] Oct 30 06:00:10.019689 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:10.019655 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" path="/var/lib/kubelet/pods/1aebf5b9-d569-40b4-8dad-672fbf43ff17/volumes" Oct 30 06:00:12.441903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.441869 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp"] Oct 30 06:00:12.442324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.442169 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" containerID="cri-o://6f4c7834647c50d87e0a4eee05e2f158c0fd41d2a569a6eac2a37b51e58a4d14" gracePeriod=30 Oct 30 06:00:12.616255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.615879 5980 generic.go:356] "Generic (PLEG): container finished" podID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerID="114a83eab9ceca15109f2720e6897bb5a951d0489eb0c4b457eff868f1f2c334" exitCode=60 Oct 30 06:00:12.616255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.615979 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerDied","Data":"114a83eab9ceca15109f2720e6897bb5a951d0489eb0c4b457eff868f1f2c334"} Oct 30 06:00:12.619141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.616505 5980 scope.go:122] "RemoveContainer" containerID="114a83eab9ceca15109f2720e6897bb5a951d0489eb0c4b457eff868f1f2c334" Oct 30 06:00:12.695285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.693535 5980 generic.go:356] "Generic (PLEG): container finished" podID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerID="435abc1ec62d857c615a719ee714b0d08af2300de21941ef4472684c9811e905" exitCode=60 Oct 30 06:00:12.695285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.693665 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerDied","Data":"435abc1ec62d857c615a719ee714b0d08af2300de21941ef4472684c9811e905"} Oct 30 06:00:12.695285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.694155 5980 scope.go:122] "RemoveContainer" containerID="435abc1ec62d857c615a719ee714b0d08af2300de21941ef4472684c9811e905" Oct 30 06:00:12.717726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.716600 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerStarted","Data":"fab51107c0af30ea2716c5f40259123b1558bd3a3040a9829a89f576c917db9a"} Oct 30 06:00:12.718988 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.718960 5980 generic.go:356] "Generic (PLEG): container finished" podID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerID="d910feb4e0f3674f847ba2f4f14a6289a7a33d96d9d99e2f8f23a66f9c369040" exitCode=60 Oct 30 06:00:12.719192 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.719032 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerDied","Data":"d910feb4e0f3674f847ba2f4f14a6289a7a33d96d9d99e2f8f23a66f9c369040"} Oct 30 06:00:12.719481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.719461 5980 scope.go:122] "RemoveContainer" containerID="d910feb4e0f3674f847ba2f4f14a6289a7a33d96d9d99e2f8f23a66f9c369040" Oct 30 06:00:12.841606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:12.841549 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" podStartSLOduration=2.446145401 podStartE2EDuration="12.841532575s" totalImagesPullingTime="10.395387174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:12.838361435 +0000 UTC m=+1666.946400464" watchObservedRunningTime="2026-10-30 06:00:12.841532575 +0000 UTC m=+1666.949571609" Oct 30 06:00:13.092760 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:13.092727 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:00:13.733464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.733430 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerStarted","Data":"5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111"} Oct 30 06:00:13.737185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.736953 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerStarted","Data":"b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d"} Oct 30 06:00:13.739509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.739481 5980 generic.go:356] "Generic (PLEG): container finished" podID="76c281b8-0fe8-4556-88ac-121455f35366" containerID="fab51107c0af30ea2716c5f40259123b1558bd3a3040a9829a89f576c917db9a" exitCode=60 Oct 30 06:00:13.739753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.739733 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerDied","Data":"fab51107c0af30ea2716c5f40259123b1558bd3a3040a9829a89f576c917db9a"} Oct 30 06:00:13.740159 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.740094 5980 scope.go:122] "RemoveContainer" containerID="fab51107c0af30ea2716c5f40259123b1558bd3a3040a9829a89f576c917db9a" Oct 30 06:00:13.746146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.746072 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerStarted","Data":"94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c"} Oct 30 06:00:13.787006 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.786943 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" podStartSLOduration=3.470066613 podStartE2EDuration="13.78692529s" totalImagesPullingTime="10.316858677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:13.783781149 +0000 UTC m=+1667.891820202" watchObservedRunningTime="2026-10-30 06:00:13.78692529 +0000 UTC m=+1667.894964324" Oct 30 06:00:13.895434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.895384 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" podStartSLOduration=3.549178291 podStartE2EDuration="13.895370236s" totalImagesPullingTime="10.346191945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:13.891089893 +0000 UTC m=+1667.999128927" watchObservedRunningTime="2026-10-30 06:00:13.895370236 +0000 UTC m=+1668.003409269" Oct 30 06:00:13.955158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:13.955079 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" podStartSLOduration=3.9223609550000003 podStartE2EDuration="13.955064702s" totalImagesPullingTime="10.032703747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:13.950314449 +0000 UTC m=+1668.058353507" watchObservedRunningTime="2026-10-30 06:00:13.955064702 +0000 UTC m=+1668.063103735" Oct 30 06:00:14.756063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.755974 5980 generic.go:356] "Generic (PLEG): container finished" podID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerID="94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c" exitCode=60 Oct 30 06:00:14.756519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.756058 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerDied","Data":"94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c"} Oct 30 06:00:14.756519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.756171 5980 scope.go:122] "RemoveContainer" containerID="d910feb4e0f3674f847ba2f4f14a6289a7a33d96d9d99e2f8f23a66f9c369040" Oct 30 06:00:14.756519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.756500 5980 scope.go:122] "RemoveContainer" containerID="94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c" Oct 30 06:00:14.756969 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:14.756944 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b_odh-ai-gateway-infra(23fe45ef-0047-4ffd-ae46-793506291ae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" Oct 30 06:00:14.758199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.758173 5980 generic.go:356] "Generic (PLEG): container finished" podID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerID="5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111" exitCode=60 Oct 30 06:00:14.758300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.758265 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerDied","Data":"5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111"} Oct 30 06:00:14.758550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.758533 5980 scope.go:122] "RemoveContainer" containerID="5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111" Oct 30 06:00:14.758874 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:14.758856 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg_odh-ai-gateway-infra(5f25d9e8-b19e-40c7-b986-33d65b0d21e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" Oct 30 06:00:14.760162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.760140 5980 generic.go:356] "Generic (PLEG): container finished" podID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerID="b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d" exitCode=60 Oct 30 06:00:14.760241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.760189 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerDied","Data":"b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d"} Oct 30 06:00:14.760435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.760422 5980 scope.go:122] "RemoveContainer" containerID="b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d" Oct 30 06:00:14.760695 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:14.760667 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk_odh-ai-gateway-infra(445d826a-7d5f-442e-9b68-7aa53a386ab9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" Oct 30 06:00:14.764151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.762406 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerStarted","Data":"a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255"} Oct 30 06:00:14.772517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.772495 5980 scope.go:122] "RemoveContainer" containerID="114a83eab9ceca15109f2720e6897bb5a951d0489eb0c4b457eff868f1f2c334" Oct 30 06:00:14.789756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:14.789688 5980 scope.go:122] "RemoveContainer" containerID="435abc1ec62d857c615a719ee714b0d08af2300de21941ef4472684c9811e905" Oct 30 06:00:15.777880 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.777845 5980 generic.go:356] "Generic (PLEG): container finished" podID="76c281b8-0fe8-4556-88ac-121455f35366" containerID="a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255" exitCode=60 Oct 30 06:00:15.778420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.777937 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerDied","Data":"a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255"} Oct 30 06:00:15.778420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.778000 5980 scope.go:122] "RemoveContainer" containerID="fab51107c0af30ea2716c5f40259123b1558bd3a3040a9829a89f576c917db9a" Oct 30 06:00:15.778420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.778295 5980 scope.go:122] "RemoveContainer" containerID="a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255" Oct 30 06:00:15.778669 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:15.778649 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889000-lkffl_odh-ai-gateway-infra(76c281b8-0fe8-4556-88ac-121455f35366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" podUID="76c281b8-0fe8-4556-88ac-121455f35366" Oct 30 06:00:15.780636 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.780620 5980 scope.go:122] "RemoveContainer" containerID="94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c" Oct 30 06:00:15.780931 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:15.780910 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b_odh-ai-gateway-infra(23fe45ef-0047-4ffd-ae46-793506291ae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" Oct 30 06:00:15.782381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.782360 5980 scope.go:122] "RemoveContainer" containerID="5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111" Oct 30 06:00:15.782841 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:15.782767 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg_odh-ai-gateway-infra(5f25d9e8-b19e-40c7-b986-33d65b0d21e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" Oct 30 06:00:15.784160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.784142 5980 scope.go:122] "RemoveContainer" containerID="b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d" Oct 30 06:00:15.784437 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:15.784416 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk_odh-ai-gateway-infra(445d826a-7d5f-442e-9b68-7aa53a386ab9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" Oct 30 06:00:15.826506 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.825887 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp"] Oct 30 06:00:15.827841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.826938 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" Oct 30 06:00:15.827841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.826961 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" Oct 30 06:00:15.827841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.827222 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" Oct 30 06:00:15.841940 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.841908 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:15.868129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.860380 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c1d09f\"" Oct 30 06:00:15.903696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:15.899315 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c1d09f-dockercfg-flxlp\"" Oct 30 06:00:16.064183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.040092 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.064183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.040440 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqml\" (UniqueName: \"kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml\") pod \"payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.110046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.110004 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp"] Oct 30 06:00:16.153165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.150160 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.153165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.150387 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqml\" (UniqueName: \"kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml\") pod \"payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.153165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.151826 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.195371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.195225 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqml\" (UniqueName: \"kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml\") pod \"payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.303133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.293633 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.676586 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:00:16.676554 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3467707d_815b_4d39_862c_783626de50ca.slice/crio-96cb05eea051206a78eb1b375e4521ecb4d97b7dff2d0c9d16e61ae8580fb919 WatchSource:0}: Error finding container 96cb05eea051206a78eb1b375e4521ecb4d97b7dff2d0c9d16e61ae8580fb919: Status 404 returned error can't find the container with id 96cb05eea051206a78eb1b375e4521ecb4d97b7dff2d0c9d16e61ae8580fb919 Oct 30 06:00:16.677966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.677939 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp"] Oct 30 06:00:16.795066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.795029 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" event={"ID":"3467707d-815b-4d39-862c-783626de50ca","Type":"ContainerStarted","Data":"eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215"} Oct 30 06:00:16.795662 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.795611 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:16.795807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.795779 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" event={"ID":"3467707d-815b-4d39-862c-783626de50ca","Type":"ContainerStarted","Data":"96cb05eea051206a78eb1b375e4521ecb4d97b7dff2d0c9d16e61ae8580fb919"} Oct 30 06:00:16.797541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.797521 5980 scope.go:122] "RemoveContainer" containerID="a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255" Oct 30 06:00:16.797929 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:16.797908 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889000-lkffl_odh-ai-gateway-infra(76c281b8-0fe8-4556-88ac-121455f35366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" podUID="76c281b8-0fe8-4556-88ac-121455f35366" Oct 30 06:00:16.843147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.843044 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" podStartSLOduration=1.843028495 podStartE2EDuration="1.843028495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:16.840923588 +0000 UTC m=+1670.948962621" watchObservedRunningTime="2026-10-30 06:00:16.843028495 +0000 UTC m=+1670.951067528" Oct 30 06:00:16.998045 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.997961 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2"] Oct 30 06:00:16.999089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.999066 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" Oct 30 06:00:16.999089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.999090 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" Oct 30 06:00:16.999400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:16.999384 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aebf5b9-d569-40b4-8dad-672fbf43ff17" containerName="istio-proxy" Oct 30 06:00:17.004463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.004435 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.044562 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.039004 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2"] Oct 30 06:00:17.065674 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.065642 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-de7242-openshift-default-dockercfg-x5rjm\"" Oct 30 06:00:17.192404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.192351 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.192404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.192407 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.192681 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.192625 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.192794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.192759 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.192860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.192834 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.192970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.192940 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mclj\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.193058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.193032 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.193225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.193184 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.193291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.193225 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295371 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295589 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295476 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295589 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295511 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295648 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295754 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295807 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295862 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mclj\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.295947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.295908 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.296035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.296019 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.296101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.296044 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.296445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.296412 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.296545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.296439 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.296648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.296620 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.296749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.296719 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.298687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.298662 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.298788 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.298767 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.324736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.324704 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.324921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.324898 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mclj\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.328750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.328727 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:17.694308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.692923 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2"] Oct 30 06:00:17.710446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.709324 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:17.710446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.709417 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:17.710446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.709492 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:17.828957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.823510 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" event={"ID":"14526aed-4138-40db-96a4-ffac615cfebd","Type":"ContainerStarted","Data":"3e52c08b8d357bc7c90a49d55135fa0f7274e504a07056139dba26fc1fea38af"} Oct 30 06:00:17.828957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.825275 5980 generic.go:356] "Generic (PLEG): container finished" podID="543da991-2779-488f-85d6-6ceeebe424fa" containerID="6f4c7834647c50d87e0a4eee05e2f158c0fd41d2a569a6eac2a37b51e58a4d14" exitCode=0 Oct 30 06:00:17.828957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.826125 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" event={"ID":"543da991-2779-488f-85d6-6ceeebe424fa","Type":"ContainerDied","Data":"6f4c7834647c50d87e0a4eee05e2f158c0fd41d2a569a6eac2a37b51e58a4d14"} Oct 30 06:00:17.828957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.826164 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" event={"ID":"543da991-2779-488f-85d6-6ceeebe424fa","Type":"ContainerDied","Data":"ac1c18256e54c990e86df36a1de928167f327a57b99d6ec9a83969dfac548e55"} Oct 30 06:00:17.828957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.826180 5980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac1c18256e54c990e86df36a1de928167f327a57b99d6ec9a83969dfac548e55" Oct 30 06:00:17.830604 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.830256 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.928635 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.928750 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.928779 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.928903 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.929020 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.929121 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.929199 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.929259 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.933132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.929296 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data\") pod \"543da991-2779-488f-85d6-6ceeebe424fa\" (UID: \"543da991-2779-488f-85d6-6ceeebe424fa\") " Oct 30 06:00:17.988143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.982769 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:17.988143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.983192 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:17.988143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.983479 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:17.988143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.987736 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:17.988143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.987761 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/543da991-2779-488f-85d6-6ceeebe424fa-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:17.988143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.987775 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.001132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:17.993186 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.018202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.013267 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.131180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.130322 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.131180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.130349 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.135857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.133259 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.138728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.137583 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:18.138728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.137882 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.138728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.138039 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf" pod "543da991-2779-488f-85d6-6ceeebe424fa" (UID: "543da991-2779-488f-85d6-6ceeebe424fa"). InnerVolumeSpecName "kube-api-access-8t4zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.233151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.233086 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/543da991-2779-488f-85d6-6ceeebe424fa-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.233151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.233136 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.233151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.233149 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t4zf\" (UniqueName: \"kubernetes.io/projected/543da991-2779-488f-85d6-6ceeebe424fa-kube-api-access-8t4zf\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.233554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.233162 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/543da991-2779-488f-85d6-6ceeebe424fa-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.896135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.895182 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp" Oct 30 06:00:18.896135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.895307 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" event={"ID":"14526aed-4138-40db-96a4-ffac615cfebd","Type":"ContainerStarted","Data":"54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c"} Oct 30 06:00:18.983720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:18.983667 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp"] Oct 30 06:00:19.012199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:19.003532 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-xxphp"] Oct 30 06:00:19.055680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:19.055579 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podStartSLOduration=3.055561283 podStartE2EDuration="3.055561283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:19.051399213 +0000 UTC m=+1673.159438246" watchObservedRunningTime="2026-10-30 06:00:19.055561283 +0000 UTC m=+1673.163600316" Oct 30 06:00:19.349927 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:19.343434 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:19.389355 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:19.379349 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:00:19.389355 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:19.379405 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 06:00:20.016761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:20.016731 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543da991-2779-488f-85d6-6ceeebe424fa" path="/var/lib/kubelet/pods/543da991-2779-488f-85d6-6ceeebe424fa/volumes" Oct 30 06:00:20.329098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:20.329067 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:00:20.329288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:20.329157 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 06:00:21.329140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:21.329068 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:00:21.329505 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:21.329155 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 06:00:22.417797 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:22.417760 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:00:22.418325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:22.417818 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 06:00:23.329140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:23.329092 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:00:23.329299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:23.329169 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 06:00:24.329188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:24.329088 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 06:00:24.329188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:24.329172 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 06:00:25.013136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:25.010072 5980 scope.go:122] "RemoveContainer" containerID="b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d" Oct 30 06:00:25.339294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:25.339265 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:25.339744 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:25.339560 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:25.340573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:25.340554 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:00:26.047026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:26.046992 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerStarted","Data":"da540f588bddea064d59e35cd54332d4d3893915e47e56e864bbd6c557e429e8"} Oct 30 06:00:27.058502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:27.058415 5980 generic.go:356] "Generic (PLEG): container finished" podID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerID="da540f588bddea064d59e35cd54332d4d3893915e47e56e864bbd6c557e429e8" exitCode=60 Oct 30 06:00:27.059371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:27.059350 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerDied","Data":"da540f588bddea064d59e35cd54332d4d3893915e47e56e864bbd6c557e429e8"} Oct 30 06:00:27.059436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:27.059425 5980 scope.go:122] "RemoveContainer" containerID="b807eb15166d4b115cf1622535dbbab6278cf08b1c38bf8b96500fa7c6af0d0d" Oct 30 06:00:27.059797 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:27.059786 5980 scope.go:122] "RemoveContainer" containerID="da540f588bddea064d59e35cd54332d4d3893915e47e56e864bbd6c557e429e8" Oct 30 06:00:27.060069 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:27.060051 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk_odh-ai-gateway-infra(445d826a-7d5f-442e-9b68-7aa53a386ab9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" Oct 30 06:00:27.102657 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:27.102619 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk"] Oct 30 06:00:28.014794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.014767 5980 scope.go:122] "RemoveContainer" containerID="94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c" Oct 30 06:00:28.014988 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.014943 5980 scope.go:122] "RemoveContainer" containerID="a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255" Oct 30 06:00:28.015947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.015770 5980 scope.go:122] "RemoveContainer" containerID="5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111" Oct 30 06:00:28.316528 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.316497 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:28.465026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.464995 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t\" (UniqueName: \"kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t\") pod \"445d826a-7d5f-442e-9b68-7aa53a386ab9\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " Oct 30 06:00:28.465399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.465380 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca\") pod \"445d826a-7d5f-442e-9b68-7aa53a386ab9\" (UID: \"445d826a-7d5f-442e-9b68-7aa53a386ab9\") " Oct 30 06:00:28.465809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.465774 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca" pod "445d826a-7d5f-442e-9b68-7aa53a386ab9" (UID: "445d826a-7d5f-442e-9b68-7aa53a386ab9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:28.466031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.466014 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/445d826a-7d5f-442e-9b68-7aa53a386ab9-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:28.467707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.467680 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t" pod "445d826a-7d5f-442e-9b68-7aa53a386ab9" (UID: "445d826a-7d5f-442e-9b68-7aa53a386ab9"). InnerVolumeSpecName "kube-api-access-4zc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:28.568033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.567977 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zc7t\" (UniqueName: \"kubernetes.io/projected/445d826a-7d5f-442e-9b68-7aa53a386ab9-kube-api-access-4zc7t\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:28.900015 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:28.899986 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:00:29.084200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.084168 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerStarted","Data":"2bbec6c33e8f8820a706ad4d6b0fe820041c2750e7699a77f6bbe637f38c39f8"} Oct 30 06:00:29.086231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.086208 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerStarted","Data":"dcf05a2e9fff4555e61f6d5186b8ac2bd6a8ee87c67673626f3e768424600e20"} Oct 30 06:00:29.087649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.087633 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" Oct 30 06:00:29.087735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.087665 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk" event={"ID":"445d826a-7d5f-442e-9b68-7aa53a386ab9","Type":"ContainerDied","Data":"03f14fc25ad231b510a86079a6c0685f1f6021d6ca1417af66c2cb08a602e334"} Oct 30 06:00:29.087735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.087712 5980 scope.go:122] "RemoveContainer" containerID="da540f588bddea064d59e35cd54332d4d3893915e47e56e864bbd6c557e429e8" Oct 30 06:00:29.090156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.090132 5980 generic.go:356] "Generic (PLEG): container finished" podID="76c281b8-0fe8-4556-88ac-121455f35366" containerID="85745ecedc27aabedab8d9b2fab0251c2caed74d8ef2ab9c977e3dbcc865ba2b" exitCode=60 Oct 30 06:00:29.090248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.090207 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerDied","Data":"85745ecedc27aabedab8d9b2fab0251c2caed74d8ef2ab9c977e3dbcc865ba2b"} Oct 30 06:00:29.090548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.090534 5980 scope.go:122] "RemoveContainer" containerID="85745ecedc27aabedab8d9b2fab0251c2caed74d8ef2ab9c977e3dbcc865ba2b" Oct 30 06:00:29.090793 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:29.090777 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889000-lkffl_odh-ai-gateway-infra(76c281b8-0fe8-4556-88ac-121455f35366)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" podUID="76c281b8-0fe8-4556-88ac-121455f35366" Oct 30 06:00:29.150818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.150793 5980 scope.go:122] "RemoveContainer" containerID="a9785aacf67290894adee4b6539a596b7f1d373c98fc4cc1bce6a06987dd6255" Oct 30 06:00:29.291288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.291245 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk"] Oct 30 06:00:29.302629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:29.302593 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-862dc2-29889000-z9wzk"] Oct 30 06:00:30.021599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.021003 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" path="/var/lib/kubelet/pods/445d826a-7d5f-442e-9b68-7aa53a386ab9/volumes" Oct 30 06:00:30.101893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.101855 5980 generic.go:356] "Generic (PLEG): container finished" podID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerID="2bbec6c33e8f8820a706ad4d6b0fe820041c2750e7699a77f6bbe637f38c39f8" exitCode=60 Oct 30 06:00:30.102087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.101916 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerDied","Data":"2bbec6c33e8f8820a706ad4d6b0fe820041c2750e7699a77f6bbe637f38c39f8"} Oct 30 06:00:30.102087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.101985 5980 scope.go:122] "RemoveContainer" containerID="94d487027b2257bd70c9fed904dd8dac4182dba177d5defcea769224110f854c" Oct 30 06:00:30.102416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.102396 5980 scope.go:122] "RemoveContainer" containerID="2bbec6c33e8f8820a706ad4d6b0fe820041c2750e7699a77f6bbe637f38c39f8" Oct 30 06:00:30.102824 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:30.102792 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b_odh-ai-gateway-infra(23fe45ef-0047-4ffd-ae46-793506291ae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" Oct 30 06:00:30.104539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.104507 5980 generic.go:356] "Generic (PLEG): container finished" podID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerID="dcf05a2e9fff4555e61f6d5186b8ac2bd6a8ee87c67673626f3e768424600e20" exitCode=60 Oct 30 06:00:30.104624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.104531 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerDied","Data":"dcf05a2e9fff4555e61f6d5186b8ac2bd6a8ee87c67673626f3e768424600e20"} Oct 30 06:00:30.110247 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.104895 5980 scope.go:122] "RemoveContainer" containerID="dcf05a2e9fff4555e61f6d5186b8ac2bd6a8ee87c67673626f3e768424600e20" Oct 30 06:00:30.110247 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:30.105257 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg_odh-ai-gateway-infra(5f25d9e8-b19e-40c7-b986-33d65b0d21e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" Oct 30 06:00:30.120368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.120282 5980 scope.go:122] "RemoveContainer" containerID="5a52291c5902e652cec96722cbf6b58a5bfafccce6e252eec44cf599ec2ef111" Oct 30 06:00:30.138548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.138526 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b"] Oct 30 06:00:30.195200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.195171 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl"] Oct 30 06:00:30.254458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.254426 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg"] Oct 30 06:00:30.374156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.374132 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:30.502880 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.502845 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4\" (UniqueName: \"kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4\") pod \"76c281b8-0fe8-4556-88ac-121455f35366\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " Oct 30 06:00:30.503079 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.503063 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca\") pod \"76c281b8-0fe8-4556-88ac-121455f35366\" (UID: \"76c281b8-0fe8-4556-88ac-121455f35366\") " Oct 30 06:00:30.503481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.503454 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca" pod "76c281b8-0fe8-4556-88ac-121455f35366" (UID: "76c281b8-0fe8-4556-88ac-121455f35366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:30.503800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.503783 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c281b8-0fe8-4556-88ac-121455f35366-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:30.505237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.505207 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4" pod "76c281b8-0fe8-4556-88ac-121455f35366" (UID: "76c281b8-0fe8-4556-88ac-121455f35366"). InnerVolumeSpecName "kube-api-access-6bmg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:30.605887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:30.605806 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmg4\" (UniqueName: \"kubernetes.io/projected/76c281b8-0fe8-4556-88ac-121455f35366-kube-api-access-6bmg4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:31.131418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.131389 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" event={"ID":"76c281b8-0fe8-4556-88ac-121455f35366","Type":"ContainerDied","Data":"b99bd986fe440996c30619e79978bc7c8e7e676da86a8d49c001bc066e753721"} Oct 30 06:00:31.131846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.131456 5980 scope.go:122] "RemoveContainer" containerID="85745ecedc27aabedab8d9b2fab0251c2caed74d8ef2ab9c977e3dbcc865ba2b" Oct 30 06:00:31.131846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.131396 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl" Oct 30 06:00:31.259547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.259509 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl"] Oct 30 06:00:31.267780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.267642 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-lkffl"] Oct 30 06:00:31.394251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.394187 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:31.397145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.397082 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:31.441199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.441168 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6\" (UniqueName: \"kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6\") pod \"23fe45ef-0047-4ffd-ae46-793506291ae4\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " Oct 30 06:00:31.441363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.441288 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca\") pod \"23fe45ef-0047-4ffd-ae46-793506291ae4\" (UID: \"23fe45ef-0047-4ffd-ae46-793506291ae4\") " Oct 30 06:00:31.441416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.441393 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca\") pod \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " Oct 30 06:00:31.441462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.441436 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7\" (UniqueName: \"kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7\") pod \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\" (UID: \"5f25d9e8-b19e-40c7-b986-33d65b0d21e8\") " Oct 30 06:00:31.441693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.441667 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca" pod "23fe45ef-0047-4ffd-ae46-793506291ae4" (UID: "23fe45ef-0047-4ffd-ae46-793506291ae4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:31.441830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.441751 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca" pod "5f25d9e8-b19e-40c7-b986-33d65b0d21e8" (UID: "5f25d9e8-b19e-40c7-b986-33d65b0d21e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:31.443851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.443822 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23fe45ef-0047-4ffd-ae46-793506291ae4-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:31.443851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.443849 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:31.444008 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.443915 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6" pod "23fe45ef-0047-4ffd-ae46-793506291ae4" (UID: "23fe45ef-0047-4ffd-ae46-793506291ae4"). InnerVolumeSpecName "kube-api-access-t8zk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:31.444343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.444323 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7" pod "5f25d9e8-b19e-40c7-b986-33d65b0d21e8" (UID: "5f25d9e8-b19e-40c7-b986-33d65b0d21e8"). InnerVolumeSpecName "kube-api-access-bwsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:31.564737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.564695 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8zk6\" (UniqueName: \"kubernetes.io/projected/23fe45ef-0047-4ffd-ae46-793506291ae4-kube-api-access-t8zk6\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:31.564737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:31.564737 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwsb7\" (UniqueName: \"kubernetes.io/projected/5f25d9e8-b19e-40c7-b986-33d65b0d21e8-kube-api-access-bwsb7\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:32.017647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.017607 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c281b8-0fe8-4556-88ac-121455f35366" path="/var/lib/kubelet/pods/76c281b8-0fe8-4556-88ac-121455f35366/volumes" Oct 30 06:00:32.146927 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.146896 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" Oct 30 06:00:32.146927 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.146903 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg" event={"ID":"5f25d9e8-b19e-40c7-b986-33d65b0d21e8","Type":"ContainerDied","Data":"3cac0d46156eb2a39daaf2f43d94096e98c048396baad4be43cf462ce54cd989"} Oct 30 06:00:32.147468 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.146972 5980 scope.go:122] "RemoveContainer" containerID="dcf05a2e9fff4555e61f6d5186b8ac2bd6a8ee87c67673626f3e768424600e20" Oct 30 06:00:32.150144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.150082 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" event={"ID":"23fe45ef-0047-4ffd-ae46-793506291ae4","Type":"ContainerDied","Data":"403b1595398d07a266c9a6c7c46be5bb88a74b77984ec2a82e4c90663fd301b1"} Oct 30 06:00:32.150256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.150157 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b" Oct 30 06:00:32.158247 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.158229 5980 scope.go:122] "RemoveContainer" containerID="2bbec6c33e8f8820a706ad4d6b0fe820041c2750e7699a77f6bbe637f38c39f8" Oct 30 06:00:32.204090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.204063 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg"] Oct 30 06:00:32.209879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.209855 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889000-s8wcg"] Oct 30 06:00:32.265059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.265024 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b"] Oct 30 06:00:32.271082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:32.270992 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889000-4wc2b"] Oct 30 06:00:34.016154 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:34.016093 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" path="/var/lib/kubelet/pods/23fe45ef-0047-4ffd-ae46-793506291ae4/volumes" Oct 30 06:00:34.016769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:34.016747 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" path="/var/lib/kubelet/pods/5f25d9e8-b19e-40c7-b986-33d65b0d21e8/volumes" Oct 30 06:00:43.090714 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:43.090672 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:00:46.437764 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.437731 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj"] Oct 30 06:00:46.438411 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438397 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438412 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438425 5980 cpu_manager.go:414] "containerMap: removing container" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438430 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438440 5980 cpu_manager.go:414] "containerMap: removing container" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438446 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438455 5980 cpu_manager.go:414] "containerMap: removing container" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" Oct 30 06:00:46.438466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438461 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438481 5980 cpu_manager.go:414] "containerMap: removing container" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438486 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438496 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438500 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438519 5980 cpu_manager.go:414] "containerMap: removing container" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438523 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438532 5980 cpu_manager.go:414] "containerMap: removing container" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438537 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438543 5980 cpu_manager.go:414] "containerMap: removing container" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438547 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438554 5980 cpu_manager.go:414] "containerMap: removing container" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438558 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438565 5980 cpu_manager.go:414] "containerMap: removing container" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438569 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438669 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438676 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438682 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.438680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438686 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:46.439277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438693 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:46.439277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438702 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:46.439277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438708 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:46.439277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438712 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:46.439277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.438718 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="543da991-2779-488f-85d6-6ceeebe424fa" containerName="istio-proxy" Oct 30 06:00:46.441961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.441946 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.465359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.465323 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj"] Oct 30 06:00:46.509339 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.509307 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c1d09f-serving-cert\"" Oct 30 06:00:46.530310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.530258 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.530514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.530492 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrmh\" (UniqueName: \"kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.530655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.530635 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.530769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.530715 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.634561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.634520 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrmh\" (UniqueName: \"kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.649258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.634627 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.649258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.634655 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.649258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.634906 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.649258 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:46.635532 5980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-serving-cert: secret "maas-api-e2e-inf-a-c1d09f-serving-cert" not found Oct 30 06:00:46.649258 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:46.635633 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls podName:3537c2ea-3c10-4101-9af9-a84b5204c016 nodeName:}" failed. No retries permitted until 2026-10-30 06:00:47.135611931 +0000 UTC m=+1701.243650945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls") pod "maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" (UID: "3537c2ea-3c10-4101-9af9-a84b5204c016") : secret "maas-api-e2e-inf-a-c1d09f-serving-cert" not found Oct 30 06:00:46.649258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.636768 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.678955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.678911 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:46.709740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:46.709658 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrmh\" (UniqueName: \"kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:47.141814 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:47.141782 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:47.144469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:47.144444 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\") pod \"maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:47.361436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:47.361394 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:48.074169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.074137 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj"] Oct 30 06:00:48.322680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.322596 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" event={"ID":"3537c2ea-3c10-4101-9af9-a84b5204c016","Type":"ContainerStarted","Data":"9d548b185d0f58735467906f2f151f48c141dc853cbf11a353979ad91d257dcd"} Oct 30 06:00:48.442279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.442194 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb"] Oct 30 06:00:48.447303 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.447271 5980 cpu_manager.go:414] "containerMap: removing container" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:48.451178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.451149 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:48.451449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.451434 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:48.451540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.451528 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:48.451852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.451832 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f25d9e8-b19e-40c7-b986-33d65b0d21e8" containerName="cleanup" Oct 30 06:00:48.451985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.451974 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="23fe45ef-0047-4ffd-ae46-793506291ae4" containerName="cleanup" Oct 30 06:00:48.452063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.452054 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="445d826a-7d5f-442e-9b68-7aa53a386ab9" containerName="cleanup" Oct 30 06:00:48.452170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.452159 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c281b8-0fe8-4556-88ac-121455f35366" containerName="cleanup" Oct 30 06:00:48.466151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.457865 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.466151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.460369 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb"] Oct 30 06:00:48.478631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.478601 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-aee070-serving-cert\"" Oct 30 06:00:48.601462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.600878 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.601462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.601138 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.601462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.601255 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.601462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.601330 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj56b\" (UniqueName: \"kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.702991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.702962 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.703360 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.703335 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.703785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.703761 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.703898 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:48.703839 5980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-serving-cert: secret "maas-api-e2e-inf-a-aee070-serving-cert" not found Oct 30 06:00:48.703969 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:48.703929 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls podName:a2d9f949-38b1-4570-a219-5cc5649f5e4f nodeName:}" failed. No retries permitted until 2026-10-30 06:00:49.2039047 +0000 UTC m=+1703.311943721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls") pod "maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" (UID: "a2d9f949-38b1-4570-a219-5cc5649f5e4f") : secret "maas-api-e2e-inf-a-aee070-serving-cert" not found Oct 30 06:00:48.703969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.703911 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj56b\" (UniqueName: \"kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.704311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.704289 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.706050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.706028 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:48.743195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:48.733456 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj56b\" (UniqueName: \"kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:49.231132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.227632 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:49.235132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.231946 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\") pod \"maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:49.331970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.331931 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" event={"ID":"3537c2ea-3c10-4101-9af9-a84b5204c016","Type":"ContainerStarted","Data":"86f308c20b62f149ed2f7d27330efae81aa1c32d57d604ae168f7c5257769f83"} Oct 30 06:00:49.332211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.332005 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:49.400516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.400451 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" podStartSLOduration=2.89925383 podStartE2EDuration="3.400434597s" totalImagesPullingTime="501.180767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:49.399149853 +0000 UTC m=+1703.507188882" watchObservedRunningTime="2026-10-30 06:00:49.400434597 +0000 UTC m=+1703.508473640" Oct 30 06:00:49.457472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.457439 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:49.793083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:49.792974 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb"] Oct 30 06:00:49.794222 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:00:49.794197 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d9f949_38b1_4570_a219_5cc5649f5e4f.slice/crio-4783488cac2ce5088c6c96915ad7b9920cdde4dc167a138b0ce1817237181cb0 WatchSource:0}: Error finding container 4783488cac2ce5088c6c96915ad7b9920cdde4dc167a138b0ce1817237181cb0: Status 404 returned error can't find the container with id 4783488cac2ce5088c6c96915ad7b9920cdde4dc167a138b0ce1817237181cb0 Oct 30 06:00:50.341829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:50.341736 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" event={"ID":"a2d9f949-38b1-4570-a219-5cc5649f5e4f","Type":"ContainerStarted","Data":"ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35"} Oct 30 06:00:50.341829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:50.341800 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" event={"ID":"a2d9f949-38b1-4570-a219-5cc5649f5e4f","Type":"ContainerStarted","Data":"4783488cac2ce5088c6c96915ad7b9920cdde4dc167a138b0ce1817237181cb0"} Oct 30 06:00:50.342367 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:50.341942 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:50.397541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:50.397487 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" podStartSLOduration=2.102685609 podStartE2EDuration="2.397472082s" totalImagesPullingTime="294.786473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:50.393929115 +0000 UTC m=+1704.501968150" watchObservedRunningTime="2026-10-30 06:00:50.397472082 +0000 UTC m=+1704.505511116" Oct 30 06:00:52.521633 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:52.521595 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn"] Oct 30 06:00:52.522143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:52.521846 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" containerID="cri-o://114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4" gracePeriod=30 Oct 30 06:00:53.090364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:53.090323 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz"] Oct 30 06:00:53.090623 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:53.090591 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" containerID="cri-o://9a377d083e767f798d9906d825d4afbd75dedc11cdacc5554222c92f591d38f3" gracePeriod=30 Oct 30 06:00:53.628307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:53.628272 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2"] Oct 30 06:00:53.628747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:53.628457 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" containerID="cri-o://8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4" gracePeriod=30 Oct 30 06:00:55.347523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:55.347493 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:00:56.185255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.185223 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd"] Oct 30 06:00:56.189961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.189940 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.214978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.214950 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd"] Oct 30 06:00:56.218948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.218921 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-387b16-serving-cert\"" Oct 30 06:00:56.280395 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.280359 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.280606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.280591 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.280693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.280679 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6c4n\" (UniqueName: \"kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.280829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.280814 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.354250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.354221 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:00:56.382773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.382743 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.382928 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.382853 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6c4n\" (UniqueName: \"kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.382992 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.382964 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.383053 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.383000 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.383799 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.383765 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.386261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.386233 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.392557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.392537 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.445231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.440273 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6c4n\" (UniqueName: \"kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n\") pod \"maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.510599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.510558 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:56.878239 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:00:56.878210 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebeaae1_7065_4605_bc17_0cf35365f795.slice/crio-6c66929250731df5ad4e8171b2d324d1c467f3b43702d83c31cb3dca65579278 WatchSource:0}: Error finding container 6c66929250731df5ad4e8171b2d324d1c467f3b43702d83c31cb3dca65579278: Status 404 returned error can't find the container with id 6c66929250731df5ad4e8171b2d324d1c467f3b43702d83c31cb3dca65579278 Oct 30 06:00:56.880052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:56.880022 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd"] Oct 30 06:00:57.400064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.400027 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" event={"ID":"debeaae1-7065-4605-bc17-0cf35365f795","Type":"ContainerStarted","Data":"fb05e3c0d84b092c49461350706de79ef7be63be889031bea54442fc9e1c5e5e"} Oct 30 06:00:57.400551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.400083 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" event={"ID":"debeaae1-7065-4605-bc17-0cf35365f795","Type":"ContainerStarted","Data":"6c66929250731df5ad4e8171b2d324d1c467f3b43702d83c31cb3dca65579278"} Oct 30 06:00:57.400551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.400263 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:00:57.454917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.454861 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" podStartSLOduration=1.081579903 podStartE2EDuration="1.454848333s" totalImagesPullingTime="373.26843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:57.451634109 +0000 UTC m=+1711.559673144" watchObservedRunningTime="2026-10-30 06:00:57.454848333 +0000 UTC m=+1711.562887401" Oct 30 06:00:57.973671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.973642 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 06:00:57.989344 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989317 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989369 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989579 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989619 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989653 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:57.989686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989680 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989798 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989719 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989798 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989755 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989819 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:57.989901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989839 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.989901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.989873 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs\") pod \"6ccf1dc3-743b-41af-9171-1a836d2f839e\" (UID: \"6ccf1dc3-743b-41af-9171-1a836d2f839e\") " Oct 30 06:00:57.990172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.990149 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:57.990452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.990430 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:57.990533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.990431 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:57.991914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.991694 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:57.992020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.991921 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:57.992020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.991938 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ccf1dc3-743b-41af-9171-1a836d2f839e-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:57.992020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.991955 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:57.992020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.991974 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:57.996438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.996407 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:57.996535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:57.996476 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:58.003237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.003214 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:58.007098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.007075 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc" pod "6ccf1dc3-743b-41af-9171-1a836d2f839e" (UID: "6ccf1dc3-743b-41af-9171-1a836d2f839e"). InnerVolumeSpecName "kube-api-access-cftsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:58.093502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.093461 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.093502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.093503 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cftsc\" (UniqueName: \"kubernetes.io/projected/6ccf1dc3-743b-41af-9171-1a836d2f839e-kube-api-access-cftsc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.093747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.093519 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.093747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.093532 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ccf1dc3-743b-41af-9171-1a836d2f839e-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.416132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.415698 5980 generic.go:356] "Generic (PLEG): container finished" podID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerID="9a377d083e767f798d9906d825d4afbd75dedc11cdacc5554222c92f591d38f3" exitCode=0 Oct 30 06:00:58.416132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.415791 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" event={"ID":"5a0a4884-4a72-4f87-b0af-af22f3123dae","Type":"ContainerDied","Data":"9a377d083e767f798d9906d825d4afbd75dedc11cdacc5554222c92f591d38f3"} Oct 30 06:00:58.417880 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.417861 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" Oct 30 06:00:58.418088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.417863 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" event={"ID":"6ccf1dc3-743b-41af-9171-1a836d2f839e","Type":"ContainerDied","Data":"114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4"} Oct 30 06:00:58.418232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.418176 5980 scope.go:122] "RemoveContainer" containerID="114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4" Oct 30 06:00:58.419220 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.419159 5980 generic.go:356] "Generic (PLEG): container finished" podID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerID="114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4" exitCode=0 Oct 30 06:00:58.419779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.419749 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn" event={"ID":"6ccf1dc3-743b-41af-9171-1a836d2f839e","Type":"ContainerDied","Data":"7c70ffbb52b19e38a2ec3aa2cb6d7929ac70d998a5cdcd348d09438017d0bc8d"} Oct 30 06:00:58.450747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.450728 5980 scope.go:122] "RemoveContainer" containerID="114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4" Oct 30 06:00:58.451024 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:58.451004 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4\": container with ID starting with 114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4 not found: ID does not exist" containerID="114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4" Oct 30 06:00:58.451093 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.451032 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4"} err="failed to get container status \"114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4\": rpc error: code = NotFound desc = could not find container \"114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4\": container with ID starting with 114fadb462b4499fc84da6b3603a4cf1b9558c0818eccc2aa4eff39d060bcad4 not found: ID does not exist" Oct 30 06:00:58.453575 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.453558 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 06:00:58.506177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506150 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.506311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506248 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.506311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506302 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.506439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506346 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.506439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506385 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.506541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506507 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.506595 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506553 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:58.506647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506615 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:58.506878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506810 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:58.507006 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506912 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.507006 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.506986 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.507134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.507099 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy\") pod \"5a0a4884-4a72-4f87-b0af-af22f3123dae\" (UID: \"5a0a4884-4a72-4f87-b0af-af22f3123dae\") " Oct 30 06:00:58.507401 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.507375 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:58.507825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.507804 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a0a4884-4a72-4f87-b0af-af22f3123dae-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.507910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.507830 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.507910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.507844 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.507910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.507856 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.508140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.508089 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:58.508894 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.508835 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:58.508983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.508957 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "kube-api-access-db65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:58.511073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.510063 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:58.511073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.511037 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy" pod "5a0a4884-4a72-4f87-b0af-af22f3123dae" (UID: "5a0a4884-4a72-4f87-b0af-af22f3123dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:58.541614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.541586 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn"] Oct 30 06:00:58.557529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.557487 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-9cwgn"] Oct 30 06:00:58.609375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.609345 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.609375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.609376 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db65t\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-kube-api-access-db65t\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.609574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.609392 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.609574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.609406 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.609574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.609419 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a0a4884-4a72-4f87-b0af-af22f3123dae-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:58.921225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:58.921163 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 06:00:59.014216 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014101 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014216 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014184 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014294 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014329 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014394 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014521 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014561 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.014884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.014809 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.015084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015061 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:59.015845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015146 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs\") pod \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\" (UID: \"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d\") " Oct 30 06:00:59.015845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015226 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:59.015845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015494 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:59.016036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015881 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.016036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015903 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.016036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.015917 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.016272 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.016249 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:59.016599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.016460 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:59.018337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.018315 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:59.018337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.018330 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:59.018841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.018819 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:59.019633 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.019606 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm" pod "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" (UID: "8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d"). InnerVolumeSpecName "kube-api-access-jtdqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:59.117919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.117890 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.117919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.117916 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.117919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.117926 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.118188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.117935 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.118188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.117944 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtdqm\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-kube-api-access-jtdqm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.118188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.117952 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:00:59.348921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.348887 5980 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:00:59.349098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.348946 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:59.427507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.427476 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" Oct 30 06:00:59.427507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.427496 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz" event={"ID":"5a0a4884-4a72-4f87-b0af-af22f3123dae","Type":"ContainerDied","Data":"5193d51a13ebca5d157b0bb638a795cd213f09e297845de5f89a23c77773d3f7"} Oct 30 06:00:59.427962 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.427559 5980 scope.go:122] "RemoveContainer" containerID="9a377d083e767f798d9906d825d4afbd75dedc11cdacc5554222c92f591d38f3" Oct 30 06:00:59.429297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.429272 5980 generic.go:356] "Generic (PLEG): container finished" podID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerID="8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4" exitCode=0 Oct 30 06:00:59.429408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.429315 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" Oct 30 06:00:59.429408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.429350 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" event={"ID":"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d","Type":"ContainerDied","Data":"8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4"} Oct 30 06:00:59.429408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.429400 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2" event={"ID":"8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d","Type":"ContainerDied","Data":"6c02bc6bcf8c8227f7d6191a76dab4447bbd4d0cf99324c67bc4f8b1863f86ff"} Oct 30 06:00:59.439049 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.439028 5980 scope.go:122] "RemoveContainer" containerID="8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4" Oct 30 06:00:59.448282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.448266 5980 scope.go:122] "RemoveContainer" containerID="8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4" Oct 30 06:00:59.448541 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:00:59.448523 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4\": container with ID starting with 8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4 not found: ID does not exist" containerID="8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4" Oct 30 06:00:59.448603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.448548 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4"} err="failed to get container status \"8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4\": rpc error: code = NotFound desc = could not find container \"8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4\": container with ID starting with 8218b1ac5e6b2965cdcf88d9874471a2b73f5d575c163e5dc33ae2b7d110a0e4 not found: ID does not exist" Oct 30 06:00:59.492164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.492102 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz"] Oct 30 06:00:59.503898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.503870 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-7wnnz"] Oct 30 06:00:59.550187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.550161 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2"] Oct 30 06:00:59.569740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:00:59.569711 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-gdgs2"] Oct 30 06:01:00.016349 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:00.016318 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" path="/var/lib/kubelet/pods/5a0a4884-4a72-4f87-b0af-af22f3123dae/volumes" Oct 30 06:01:00.016854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:00.016839 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" path="/var/lib/kubelet/pods/6ccf1dc3-743b-41af-9171-1a836d2f839e/volumes" Oct 30 06:01:00.017303 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:00.017276 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" path="/var/lib/kubelet/pods/8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d/volumes" Oct 30 06:01:03.425841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.425812 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:01:03.465969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.465933 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz"] Oct 30 06:01:03.466735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466717 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" Oct 30 06:01:03.466735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466735 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" Oct 30 06:01:03.466886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466767 5980 cpu_manager.go:414] "containerMap: removing container" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" Oct 30 06:01:03.466886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466773 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" Oct 30 06:01:03.466886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466794 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" Oct 30 06:01:03.466886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466799 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" Oct 30 06:01:03.467086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466911 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ccf1dc3-743b-41af-9171-1a836d2f839e" containerName="istio-proxy" Oct 30 06:01:03.467086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466923 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0a4884-4a72-4f87-b0af-af22f3123dae" containerName="istio-proxy" Oct 30 06:01:03.467086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.466934 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9e93d3-9828-4fce-ad65-9a3e81ce9c8d" containerName="istio-proxy" Oct 30 06:01:03.471584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.471558 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.516534 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.515133 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz"] Oct 30 06:01:03.516534 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.515617 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7df9fa-openshift-default-dockercfg-whvxb\"" Oct 30 06:01:03.589126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.588633 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.588755 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.588991 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.589029 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29hn\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.589099 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.589207 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.589245 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.589346 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.589471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.589439 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693307 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693384 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693621 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693653 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29hn\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693694 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693722 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693768 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693838 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.693934 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.694823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.694474 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.701374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.695033 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.711132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.703770 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.711132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.704397 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.714979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.711976 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.714979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.714486 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.782074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.782042 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29hn\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.795128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.793890 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:03.805138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:03.802051 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:04.088615 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:04.088573 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:04.431396 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:04.431368 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz"] Oct 30 06:01:04.432340 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:04.432307 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073b49be_33b8_4fa0_8d02_599933418225.slice/crio-ed53f5c57983a56c1ebd96ada0224b06ee31df8e1d916fbe2161cba97a6165f1 WatchSource:0}: Error finding container ed53f5c57983a56c1ebd96ada0224b06ee31df8e1d916fbe2161cba97a6165f1: Status 404 returned error can't find the container with id ed53f5c57983a56c1ebd96ada0224b06ee31df8e1d916fbe2161cba97a6165f1 Oct 30 06:01:04.435185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:04.435152 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:04.435271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:04.435221 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:04.435271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:04.435251 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:04.488636 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:04.488600 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" event={"ID":"073b49be-33b8-4fa0-8d02-599933418225","Type":"ContainerStarted","Data":"ed53f5c57983a56c1ebd96ada0224b06ee31df8e1d916fbe2161cba97a6165f1"} Oct 30 06:01:05.500805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:05.500758 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" event={"ID":"073b49be-33b8-4fa0-8d02-599933418225","Type":"ContainerStarted","Data":"193c4f99ced2b86f73c51a5def782b9684d50eaa5d9a64a136a60c9b45297088"} Oct 30 06:01:05.589731 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:05.589664 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podStartSLOduration=2.589644133 podStartE2EDuration="2.589644133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:05.57364356 +0000 UTC m=+1719.681682594" watchObservedRunningTime="2026-10-30 06:01:05.589644133 +0000 UTC m=+1719.697683167" Oct 30 06:01:06.088989 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:06.088960 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:06.095203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:06.095176 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:06.507134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:06.507031 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:06.508164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:06.508145 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:01:10.269405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.269371 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm"] Oct 30 06:01:10.274282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.274261 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.307883 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.307848 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7df9fa\"" Oct 30 06:01:10.313726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.313700 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm"] Oct 30 06:01:10.401297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.401264 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp"] Oct 30 06:01:10.443136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.438140 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.494517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.472789 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7df9fa-dockercfg-mxqkn\"" Oct 30 06:01:10.494724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.482224 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st4n\" (UniqueName: \"kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n\") pod \"payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.518135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.495003 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.535181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.529000 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75dx\" (UniqueName: \"kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx\") pod \"payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.535181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.529175 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.567220 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.567179 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp"] Oct 30 06:01:10.634501 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.634467 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75dx\" (UniqueName: \"kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx\") pod \"payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.636654 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.636623 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.643907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.642628 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st4n\" (UniqueName: \"kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n\") pod \"payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.643907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.642962 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.648211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.645874 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.648211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.646542 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.684161 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.682333 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75dx\" (UniqueName: \"kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx\") pod \"payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.684161 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.682705 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:10.702648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.702593 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st4n\" (UniqueName: \"kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n\") pod \"payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:10.889458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:10.889428 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:11.033393 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:11.033162 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode628adbf_9199_4539_8bed_1f4e186922e3.slice/crio-c9ecb630eb933dd85326026f62d9a1ad4c196252a24b3c497070f8d9c62de550 WatchSource:0}: Error finding container c9ecb630eb933dd85326026f62d9a1ad4c196252a24b3c497070f8d9c62de550: Status 404 returned error can't find the container with id c9ecb630eb933dd85326026f62d9a1ad4c196252a24b3c497070f8d9c62de550 Oct 30 06:01:11.036151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.036084 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp"] Oct 30 06:01:11.238932 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:11.238894 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb298cb49_585f_46f9_be12_67cd92b13c06.slice/crio-1d6f166903579a9d3cddc46aafff4e41cd80529e3655074ec316031e983a20fb WatchSource:0}: Error finding container 1d6f166903579a9d3cddc46aafff4e41cd80529e3655074ec316031e983a20fb: Status 404 returned error can't find the container with id 1d6f166903579a9d3cddc46aafff4e41cd80529e3655074ec316031e983a20fb Oct 30 06:01:11.239214 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.239194 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm"] Oct 30 06:01:11.641470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.641423 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" event={"ID":"b298cb49-585f-46f9-be12-67cd92b13c06","Type":"ContainerStarted","Data":"eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391"} Oct 30 06:01:11.641958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.641497 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" event={"ID":"b298cb49-585f-46f9-be12-67cd92b13c06","Type":"ContainerStarted","Data":"1d6f166903579a9d3cddc46aafff4e41cd80529e3655074ec316031e983a20fb"} Oct 30 06:01:11.641958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.641673 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:11.643302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.643275 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" event={"ID":"e628adbf-9199-4539-8bed-1f4e186922e3","Type":"ContainerStarted","Data":"c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3"} Oct 30 06:01:11.643457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.643323 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" event={"ID":"e628adbf-9199-4539-8bed-1f4e186922e3","Type":"ContainerStarted","Data":"c9ecb630eb933dd85326026f62d9a1ad4c196252a24b3c497070f8d9c62de550"} Oct 30 06:01:11.643457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.643348 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:11.694030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.693988 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" podStartSLOduration=1.693976171 podStartE2EDuration="1.693976171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:11.693502621 +0000 UTC m=+1725.801541653" watchObservedRunningTime="2026-10-30 06:01:11.693976171 +0000 UTC m=+1725.802015236" Oct 30 06:01:11.741985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:11.741941 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" podStartSLOduration=1.741928066 podStartE2EDuration="1.741928066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:11.741658197 +0000 UTC m=+1725.849697230" watchObservedRunningTime="2026-10-30 06:01:11.741928066 +0000 UTC m=+1725.849967106" Oct 30 06:01:13.111018 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:01:13.110976 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:01:16.621551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.621185 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n"] Oct 30 06:01:16.626357 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.626325 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.661743 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.661721 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3cc570-openshift-default-dockercfg-svxdq\"" Oct 30 06:01:16.670750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.670726 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.670879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.670757 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.670879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.670861 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.670993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.670970 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.671076 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.671060 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.671166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.671094 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.671630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.671317 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.671630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.671478 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttgq\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.671630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.671542 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.675648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.675625 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n"] Oct 30 06:01:16.773677 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.773639 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.773812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.773754 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.773879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.773860 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.773933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.773890 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774048 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774018 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774141 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttgq\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774188 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774275 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774301 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774402 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774654 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.774937 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.774787 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.775052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.775025 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.775191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.775141 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.776485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.776462 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.776596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.776572 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.802448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.802403 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.802448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.802415 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttgq\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:16.943776 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:16.943694 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:17.458251 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:17.458218 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b028d8a_67dc_41fa_ba09_b4c59b3d7777.slice/crio-a80dc21a199d05973d53e854722c8634dec42ca091a915d8a643613a589ba85a WatchSource:0}: Error finding container a80dc21a199d05973d53e854722c8634dec42ca091a915d8a643613a589ba85a: Status 404 returned error can't find the container with id a80dc21a199d05973d53e854722c8634dec42ca091a915d8a643613a589ba85a Oct 30 06:01:17.458251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.458235 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n"] Oct 30 06:01:17.461611 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.461577 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:17.461699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.461656 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:17.461738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.461710 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:17.714043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.713883 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" event={"ID":"9b028d8a-67dc-41fa-ba09-b4c59b3d7777","Type":"ContainerStarted","Data":"c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832"} Oct 30 06:01:17.714043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.713947 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" event={"ID":"9b028d8a-67dc-41fa-ba09-b4c59b3d7777","Type":"ContainerStarted","Data":"a80dc21a199d05973d53e854722c8634dec42ca091a915d8a643613a589ba85a"} Oct 30 06:01:17.764640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.764589 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podStartSLOduration=1.764571311 podStartE2EDuration="1.764571311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:17.762975414 +0000 UTC m=+1731.871014448" watchObservedRunningTime="2026-10-30 06:01:17.764571311 +0000 UTC m=+1731.872610344" Oct 30 06:01:17.947353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:17.947238 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:18.954068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:18.954034 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:19.728021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:19.727992 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:19.728996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:19.728978 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:01:22.481546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.481511 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7"] Oct 30 06:01:22.489807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.489774 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:22.513456 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.511873 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7"] Oct 30 06:01:22.580475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.580446 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3cc570-dockercfg-m9gn7\"" Oct 30 06:01:22.612130 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.580732 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3cc570\"" Oct 30 06:01:22.646397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.643442 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rcdq\" (UniqueName: \"kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq\") pod \"payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:22.646397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.643692 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:22.757649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.757559 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:01:22.780061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.780031 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:01:22.810131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.799727 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rcdq\" (UniqueName: \"kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq\") pod \"payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:22.810131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.799820 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:22.829134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.823204 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:22.859714 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:22.844145 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rcdq\" (UniqueName: \"kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq\") pod \"payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:23.120175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:23.120135 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:23.462183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:23.462155 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7"] Oct 30 06:01:23.462667 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:23.462642 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968c465c_a936_4ca0_acd4_10c7aa3dc2fd.slice/crio-6f72d624c127b6ac3e81bce69e42ba6eb3965e2a6262c86bdded1e91dffb90e5 WatchSource:0}: Error finding container 6f72d624c127b6ac3e81bce69e42ba6eb3965e2a6262c86bdded1e91dffb90e5: Status 404 returned error can't find the container with id 6f72d624c127b6ac3e81bce69e42ba6eb3965e2a6262c86bdded1e91dffb90e5 Oct 30 06:01:23.855514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:23.855481 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" event={"ID":"968c465c-a936-4ca0-acd4-10c7aa3dc2fd","Type":"ContainerStarted","Data":"108ecc79d3669b810d39065ae4e8a422974549f6689d1fc1ea53b7dfb0f5143f"} Oct 30 06:01:23.855909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:23.855542 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" event={"ID":"968c465c-a936-4ca0-acd4-10c7aa3dc2fd","Type":"ContainerStarted","Data":"6f72d624c127b6ac3e81bce69e42ba6eb3965e2a6262c86bdded1e91dffb90e5"} Oct 30 06:01:23.855909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:23.855570 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:23.915738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:23.915694 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" podStartSLOduration=1.915681138 podStartE2EDuration="1.915681138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:23.914751381 +0000 UTC m=+1738.022790414" watchObservedRunningTime="2026-10-30 06:01:23.915681138 +0000 UTC m=+1738.023720170" Oct 30 06:01:24.302582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.302482 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs"] Oct 30 06:01:24.307036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.307011 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.398600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.396171 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-862dc2-openshift-default-dockercfg-8mcqs\"" Oct 30 06:01:24.399243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.399205 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs"] Oct 30 06:01:24.469055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469017 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469081 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469170 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvft\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469208 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469235 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469258 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469318 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469383 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.469475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.469439 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.570944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.570915 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.570997 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571240 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571319 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571305 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571382 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvft\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571412 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571430 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571556 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571490 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571556 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571529 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571629 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.571800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571770 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.572032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.571990 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.572032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.572046 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.572032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.572074 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.573901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.573875 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.574548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.574525 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.604564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.604491 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvft\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.604675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.604570 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.622418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.622039 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:24.966641 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:24.966594 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aa81c6_2469_4e68_abed_712ffa461dd8.slice/crio-4e5528257e409b3e4d95141f4d15f72a0bb0cf283c6b71ada7febc2b2cdebddc WatchSource:0}: Error finding container 4e5528257e409b3e4d95141f4d15f72a0bb0cf283c6b71ada7febc2b2cdebddc: Status 404 returned error can't find the container with id 4e5528257e409b3e4d95141f4d15f72a0bb0cf283c6b71ada7febc2b2cdebddc Oct 30 06:01:24.969277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.969257 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs"] Oct 30 06:01:24.970466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.970424 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:24.970527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.970511 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:24.970577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:24.970557 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:25.899410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:25.899358 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" event={"ID":"91aa81c6-2469-4e68-abed-712ffa461dd8","Type":"ContainerStarted","Data":"0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7"} Oct 30 06:01:25.899618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:25.899422 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" event={"ID":"91aa81c6-2469-4e68-abed-712ffa461dd8","Type":"ContainerStarted","Data":"4e5528257e409b3e4d95141f4d15f72a0bb0cf283c6b71ada7febc2b2cdebddc"} Oct 30 06:01:25.996681 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:25.996612 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podStartSLOduration=1.996589476 podStartE2EDuration="1.996589476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:25.97560076 +0000 UTC m=+1740.083639793" watchObservedRunningTime="2026-10-30 06:01:25.996589476 +0000 UTC m=+1740.104628511" Oct 30 06:01:26.623285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:26.623250 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:26.624727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:26.624690 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:01:26.624879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:26.624741 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 06:01:27.623358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:27.623326 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:01:27.623751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:27.623380 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 06:01:28.623300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:28.623261 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:01:28.623517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:28.623326 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 06:01:29.642131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:29.626187 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:01:29.642131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:29.626250 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 06:01:30.625131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:30.622512 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:01:30.625131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:30.622569 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 06:01:31.627715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:31.627440 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 06:01:31.627715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:31.627498 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 06:01:32.688580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:32.688549 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:32.689363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:32.689276 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:32.690165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:32.690140 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:01:34.871167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:34.871136 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:01:43.089761 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:01:43.089719 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:01:44.107399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.107362 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc"] Oct 30 06:01:44.113389 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.113368 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.146559 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.146531 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc"] Oct 30 06:01:44.167945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.167918 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-387b16-openshift-default-dockercfg-hxplc\"" Oct 30 06:01:44.188648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.188614 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.188840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.188661 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.188840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.188774 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.188840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.188825 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.189018 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.188877 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.189018 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.188908 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.189179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.189033 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.189179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.189154 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.189304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.189281 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86b2\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290265 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290226 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290320 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290407 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86b2\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290558 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290593 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290639 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290667 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290677 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290875 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290770 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.290875 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290803 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.291004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.290968 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.291065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.291040 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.291270 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.291251 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.291442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.291422 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.293553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.293525 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.293644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.293560 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.322596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.322569 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.322985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.322960 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86b2\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.431886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.431770 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:44.818081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.818038 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc"] Oct 30 06:01:44.818231 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:44.818198 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc203a63b_accb_4462_88a0_1b3dc81fa87b.slice/crio-d0adf62fc2c416fca90400be04cc57ab126c0908e50d23b9e0e65ca609d62ed7 WatchSource:0}: Error finding container d0adf62fc2c416fca90400be04cc57ab126c0908e50d23b9e0e65ca609d62ed7: Status 404 returned error can't find the container with id d0adf62fc2c416fca90400be04cc57ab126c0908e50d23b9e0e65ca609d62ed7 Oct 30 06:01:44.821158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.821121 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:44.821247 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.821196 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:44.821247 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:44.821225 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:45.238328 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:45.238288 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" event={"ID":"c203a63b-accb-4462-88a0-1b3dc81fa87b","Type":"ContainerStarted","Data":"d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9"} Oct 30 06:01:45.238749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:45.238346 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" event={"ID":"c203a63b-accb-4462-88a0-1b3dc81fa87b","Type":"ContainerStarted","Data":"d0adf62fc2c416fca90400be04cc57ab126c0908e50d23b9e0e65ca609d62ed7"} Oct 30 06:01:45.321334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:45.320315 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podStartSLOduration=1.320295311 podStartE2EDuration="1.320295311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:45.319791859 +0000 UTC m=+1759.427830894" watchObservedRunningTime="2026-10-30 06:01:45.320295311 +0000 UTC m=+1759.428334348" Oct 30 06:01:45.439132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:45.439085 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:45.440138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:45.440096 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:45.440316 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:45.440292 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:46.205632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.205598 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4"] Oct 30 06:01:46.210694 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.210669 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.241405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.241377 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4"] Oct 30 06:01:46.250738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.250716 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.250867 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.250780 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.250960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.250932 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.251024 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.250987 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.251132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.251088 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9c2j\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.251191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.251182 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.251270 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.251256 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.251339 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.251326 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.251374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.251365 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.298414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.298381 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7ad9ad-openshift-default-dockercfg-kg24d\"" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.370439 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.370566 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.370667 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.370725 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.370834 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c2j\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.370909 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.371017 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.371185 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.371255 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.372133 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.379162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.372372 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.468183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.438465 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.468183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.438995 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.468183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.439260 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.500138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.488230 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:46.500138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.488284 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:46.520157 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.513449 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.723424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.723350 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c2j\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.727849 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.727795 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.729593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.729573 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:46.843341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:46.843086 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:47.327852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:47.327203 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4"] Oct 30 06:01:47.340189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:47.340146 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:47.340631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:47.340605 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:47.340757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:47.340653 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:47.433231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:47.433198 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:47.433416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:47.433253 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:48.348057 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.348012 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" event={"ID":"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f","Type":"ContainerStarted","Data":"39ed0ad813770ba42c775c05df852a1c62e3a6e5bba9c4eef9fa1d0840daa8ad"} Oct 30 06:01:48.348573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.348083 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" event={"ID":"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f","Type":"ContainerStarted","Data":"aeb53b8dbb435988a3468b7558095078dce2b4a0942707179a4689a0a83b5c91"} Oct 30 06:01:48.433039 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.433010 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:48.433221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.433058 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:48.447700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.447651 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podStartSLOduration=2.447635468 podStartE2EDuration="2.447635468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:48.447163087 +0000 UTC m=+1762.555202121" watchObservedRunningTime="2026-10-30 06:01:48.447635468 +0000 UTC m=+1762.555674501" Oct 30 06:01:48.844031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.844001 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:48.850239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:48.850214 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:49.355972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:49.355946 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:49.356999 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:49.356977 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:01:49.432347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:49.432316 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:49.432533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:49.432365 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:50.016963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.016935 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln"] Oct 30 06:01:50.021630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.021611 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.038398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.038372 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-62316c-dockercfg-gwtgv\"" Oct 30 06:01:50.039552 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.039530 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln"] Oct 30 06:01:50.044415 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.044380 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmbk\" (UniqueName: \"kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk\") pod \"payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.044776 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.044753 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.045166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.045146 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-62316c\"" Oct 30 06:01:50.455493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.455457 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.456721 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.456464 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmbk\" (UniqueName: \"kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk\") pod \"payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.494382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.470591 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.523134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.502864 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:50.523464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.523430 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:50.537476 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.537445 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb"] Oct 30 06:01:50.547065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.547030 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmbk\" (UniqueName: \"kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk\") pod \"payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.548628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.548604 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb"] Oct 30 06:01:50.548808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.548734 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.550752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.550732 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4"] Oct 30 06:01:50.554925 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.554908 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4"] Oct 30 06:01:50.555019 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.554977 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.638454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.638423 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:50.646563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.646544 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1d09f-openshift-default-dockercfg-gpb22\"" Oct 30 06:01:50.703241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.703178 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.703839 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.703527 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6snf\" (UniqueName: \"kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf\") pod \"payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.704035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.704016 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.704253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.704237 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.704514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.704450 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.705064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.705011 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.705285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.705269 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.705449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.705434 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2qr\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.705870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.705614 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.706012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.705997 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.706199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.706184 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823239 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823293 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823326 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2qr\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823375 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823404 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823442 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823682 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823757 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6snf\" (UniqueName: \"kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf\") pod \"payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823794 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823845 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.823913 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.836979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.824868 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.871820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.851182 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.871820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.852004 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.871820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.852528 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.871820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.852599 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.871820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.852861 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.871820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.858955 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.900017 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.897594 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.910983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.910686 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.910983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.910898 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6snf\" (UniqueName: \"kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf\") pod \"payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.910983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.910949 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2qr\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:50.910983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.910981 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:50.917165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:50.916932 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:51.026364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.024825 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln"] Oct 30 06:01:51.035754 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:51.034675 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc2285c_ef59_4a3b_a138_d7018678fe3c.slice/crio-0e6982c46b102264b6e242536c4e8fbf458d165ab2ac65d76f00837cdbe49218 WatchSource:0}: Error finding container 0e6982c46b102264b6e242536c4e8fbf458d165ab2ac65d76f00837cdbe49218: Status 404 returned error can't find the container with id 0e6982c46b102264b6e242536c4e8fbf458d165ab2ac65d76f00837cdbe49218 Oct 30 06:01:51.324687 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:51.324638 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3674ab80_d2a1_4abe_8f8e_02dbe72b344a.slice/crio-8c398073b209d035108c550bc543e21e44ca9dd79b71989f0c00c03dea04abad WatchSource:0}: Error finding container 8c398073b209d035108c550bc543e21e44ca9dd79b71989f0c00c03dea04abad: Status 404 returned error can't find the container with id 8c398073b209d035108c550bc543e21e44ca9dd79b71989f0c00c03dea04abad Oct 30 06:01:51.326318 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:51.326287 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d9e535_6153_49bb_8b17_830f60a9a3ec.slice/crio-5bf925e3aa9aec080d506f547576f11c092b46a9004bb9ef024f9c85076c7b27 WatchSource:0}: Error finding container 5bf925e3aa9aec080d506f547576f11c092b46a9004bb9ef024f9c85076c7b27: Status 404 returned error can't find the container with id 5bf925e3aa9aec080d506f547576f11c092b46a9004bb9ef024f9c85076c7b27 Oct 30 06:01:51.328254 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.328232 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb"] Oct 30 06:01:51.332212 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.332188 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4"] Oct 30 06:01:51.332827 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.332799 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.332915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.332856 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.332915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.332888 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.433947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.433921 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 06:01:51.434058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.433969 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 06:01:51.552700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.552656 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" event={"ID":"e1d9e535-6153-49bb-8b17-830f60a9a3ec","Type":"ContainerStarted","Data":"3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5"} Oct 30 06:01:51.553195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.552719 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" event={"ID":"e1d9e535-6153-49bb-8b17-830f60a9a3ec","Type":"ContainerStarted","Data":"5bf925e3aa9aec080d506f547576f11c092b46a9004bb9ef024f9c85076c7b27"} Oct 30 06:01:51.555035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.555006 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" event={"ID":"3674ab80-d2a1-4abe-8f8e-02dbe72b344a","Type":"ContainerStarted","Data":"6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14"} Oct 30 06:01:51.555177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.555056 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" event={"ID":"3674ab80-d2a1-4abe-8f8e-02dbe72b344a","Type":"ContainerStarted","Data":"8c398073b209d035108c550bc543e21e44ca9dd79b71989f0c00c03dea04abad"} Oct 30 06:01:51.555797 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.555778 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:01:51.558275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.558252 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" event={"ID":"bdc2285c-ef59-4a3b-a138-d7018678fe3c","Type":"ContainerStarted","Data":"a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e"} Oct 30 06:01:51.558380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.558292 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" event={"ID":"bdc2285c-ef59-4a3b-a138-d7018678fe3c","Type":"ContainerStarted","Data":"0e6982c46b102264b6e242536c4e8fbf458d165ab2ac65d76f00837cdbe49218"} Oct 30 06:01:51.558380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.558346 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:01:51.620559 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.620459 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podStartSLOduration=1.620438898 podStartE2EDuration="1.620438898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:51.616558743 +0000 UTC m=+1765.724597778" watchObservedRunningTime="2026-10-30 06:01:51.620438898 +0000 UTC m=+1765.728477933" Oct 30 06:01:51.749806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.749725 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" podStartSLOduration=2.749706121 podStartE2EDuration="2.749706121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:51.744366505 +0000 UTC m=+1765.852405539" watchObservedRunningTime="2026-10-30 06:01:51.749706121 +0000 UTC m=+1765.857745156" Oct 30 06:01:51.751861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.751808 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" podStartSLOduration=1.751792746 podStartE2EDuration="1.751792746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:51.685944766 +0000 UTC m=+1765.793983821" watchObservedRunningTime="2026-10-30 06:01:51.751792746 +0000 UTC m=+1765.859831780" Oct 30 06:01:51.927640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:51.925208 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:52.452179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:52.452150 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:52.452802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:52.452737 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:52.453748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:52.453723 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:01:52.922199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:52.922162 5980 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:01:52.922614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:52.922215 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 06:01:53.552281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.552246 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk"] Oct 30 06:01:53.557037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.557009 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:53.609765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.602506 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk"] Oct 30 06:01:53.610006 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.603828 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95q6v\" (UniqueName: \"kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v\") pod \"payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:53.610211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.610191 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:53.610736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.610698 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7ad9ad-dockercfg-xrsps\"" Oct 30 06:01:53.610934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.610918 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7ad9ad\"" Oct 30 06:01:53.618129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.616536 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z"] Oct 30 06:01:53.637658 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.637637 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:53.652248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.652221 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z"] Oct 30 06:01:53.765609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.765521 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95q6v\" (UniqueName: \"kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v\") pod \"payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:53.778132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.774679 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:53.778132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.774758 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:53.778132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.774943 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2x5\" (UniqueName: \"kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5\") pod \"payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:53.803186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.780148 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:53.891133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.865137 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95q6v\" (UniqueName: \"kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v\") pod \"payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:54.056692 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.929569 5980 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.063941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.057333 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 06:01:54.066335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.931064 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:54.067904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:53.930210 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:54.068213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.068191 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2x5\" (UniqueName: \"kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5\") pod \"payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:54.113897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.113672 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2x5\" (UniqueName: \"kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5\") pod \"payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:54.216258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.216226 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:54.395365 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.395327 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:54.635430 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.635010 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk"] Oct 30 06:01:54.647078 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.647051 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" event={"ID":"cbdf8e63-5588-49de-a008-b864cd633447","Type":"ContainerStarted","Data":"97337af7f62ad1416ff2d634b40b56211e90e98942244345d928c3553d00a069"} Oct 30 06:01:54.792273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.792245 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z"] Oct 30 06:01:54.795026 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:01:54.795005 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef86243d_038b_45e6_95ee_aa4a2f279bde.slice/crio-3f02244c1cc6c814943756bf9ef853ad000befbfe087f5eefb703799d49c37dd WatchSource:0}: Error finding container 3f02244c1cc6c814943756bf9ef853ad000befbfe087f5eefb703799d49c37dd: Status 404 returned error can't find the container with id 3f02244c1cc6c814943756bf9ef853ad000befbfe087f5eefb703799d49c37dd Oct 30 06:01:54.917916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.917888 5980 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.918048 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:54.917937 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 06:01:55.660158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.655964 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" event={"ID":"cbdf8e63-5588-49de-a008-b864cd633447","Type":"ContainerStarted","Data":"971afb3d24fd57438cc910caf3b741fadba81331ef1b8579db442d2e312c0bd8"} Oct 30 06:01:55.660158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.656036 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:01:55.660158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.659844 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" event={"ID":"ef86243d-038b-45e6-95ee-aa4a2f279bde","Type":"ContainerStarted","Data":"3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85"} Oct 30 06:01:55.660158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.659876 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" event={"ID":"ef86243d-038b-45e6-95ee-aa4a2f279bde","Type":"ContainerStarted","Data":"3f02244c1cc6c814943756bf9ef853ad000befbfe087f5eefb703799d49c37dd"} Oct 30 06:01:55.660158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.659938 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:01:55.726099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.726037 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" podStartSLOduration=2.7260233940000003 podStartE2EDuration="2.726023394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:55.722571597 +0000 UTC m=+1769.830610629" watchObservedRunningTime="2026-10-30 06:01:55.726023394 +0000 UTC m=+1769.834062802" Oct 30 06:01:55.918934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.918852 5980 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:01:55.918934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:55.918901 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 06:01:56.917471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:56.917426 5980 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:01:56.917859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:56.917497 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 06:01:57.917626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:57.917591 5980 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 06:01:57.918242 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:57.917659 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 06:01:58.923004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:58.922971 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:58.923561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:58.923476 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:58.924257 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:58.924236 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:01:58.991733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:01:58.991674 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" podStartSLOduration=5.991654776 podStartE2EDuration="5.991654776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:55.790593648 +0000 UTC m=+1769.898632684" watchObservedRunningTime="2026-10-30 06:01:58.991654776 +0000 UTC m=+1773.099693815" Oct 30 06:02:02.590998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:02.590969 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:02:03.636725 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:03.636679 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:02:06.702703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:06.702666 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:02:06.703653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:06.703628 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:02:13.096077 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:02:13.096034 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:02:32.114030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:32.113991 5980 generic.go:356] "Generic (PLEG): container finished" podID="073b49be-33b8-4fa0-8d02-599933418225" containerID="193c4f99ced2b86f73c51a5def782b9684d50eaa5d9a64a136a60c9b45297088" exitCode=0 Oct 30 06:02:32.114553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:32.114035 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" event={"ID":"073b49be-33b8-4fa0-8d02-599933418225","Type":"ContainerDied","Data":"193c4f99ced2b86f73c51a5def782b9684d50eaa5d9a64a136a60c9b45297088"} Oct 30 06:02:32.114631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:32.114613 5980 scope.go:122] "RemoveContainer" containerID="193c4f99ced2b86f73c51a5def782b9684d50eaa5d9a64a136a60c9b45297088" Oct 30 06:02:32.116633 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:32.116596 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:32.116860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:32.116833 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:32.116972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:32.116897 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:33.167869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:33.167824 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" event={"ID":"073b49be-33b8-4fa0-8d02-599933418225","Type":"ContainerStarted","Data":"22c0c7a8c47f8693bba21556cfd43582aa679b4a4022abe7f033d52f2c7cd610"} Oct 30 06:02:34.096145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:34.095775 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:02:34.098492 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:34.098458 5980 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:02:34.098639 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:34.098513 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 06:02:35.101171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:35.094142 5980 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:02:35.101171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:35.094210 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 06:02:36.093191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:36.089502 5980 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:02:36.093191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:36.089561 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 06:02:36.507580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:36.507487 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:02:37.095805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:37.095770 5980 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:02:37.097683 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:37.097650 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 06:02:38.089182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:38.089098 5980 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:02:38.090373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:38.089984 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 06:02:39.089735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:39.089697 5980 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 06:02:39.090207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:39.089780 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 06:02:40.096476 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:40.096442 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:02:40.097909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:40.097890 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:02:43.091837 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:02:43.091798 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:02:59.507239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:59.507152 5980 generic.go:356] "Generic (PLEG): container finished" podID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerID="c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832" exitCode=0 Oct 30 06:02:59.507702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:59.507223 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" event={"ID":"9b028d8a-67dc-41fa-ba09-b4c59b3d7777","Type":"ContainerDied","Data":"c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832"} Oct 30 06:02:59.507880 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:59.507864 5980 scope.go:122] "RemoveContainer" containerID="c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832" Oct 30 06:02:59.510099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:59.510068 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:59.510229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:59.510200 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:59.510296 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:02:59.510253 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:00.307059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:00.307014 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz"] Oct 30 06:03:00.341375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:00.341341 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr"] Oct 30 06:03:00.551359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:00.551325 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" event={"ID":"9b028d8a-67dc-41fa-ba09-b4c59b3d7777","Type":"ContainerStarted","Data":"cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9"} Oct 30 06:03:00.958135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:00.957451 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:03:00.967174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:00.962250 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:03:00.967174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:00.962310 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:03:01.948024 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:01.947984 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:03:01.948498 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:01.948042 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:03:02.947467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:02.947434 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:03:02.947648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:02.947490 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:03:03.948381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:03.948343 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:03:03.948796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:03.948418 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:03:04.728069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:04.728041 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:03:04.948478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:04.948446 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:03:04.948889 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:04.948510 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:03:05.308379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:05.308325 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerName="payload-processing" containerID="cri-o://b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1" gracePeriod=25 Oct 30 06:03:05.342001 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:05.341948 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" podUID="998663cb-bb11-4252-b354-ac825b8a096f" containerName="payload-pre-processing" containerID="cri-o://6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1" gracePeriod=25 Oct 30 06:03:05.948012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:05.947973 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:03:05.948255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:05.948042 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:03:06.954991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:06.954953 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:03:06.956167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:06.956144 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:03:10.590952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:10.590910 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:03:12.758485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:12.758439 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" podUID="998663cb-bb11-4252-b354-ac825b8a096f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:03:13.099296 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:03:13.099253 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:03:20.591239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:20.591202 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:03:22.760411 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:22.760370 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" podUID="998663cb-bb11-4252-b354-ac825b8a096f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:03:26.037896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.037872 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 06:03:26.041946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.041925 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 06:03:26.094033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.093991 5980 generic.go:356] "Generic (PLEG): container finished" podID="998663cb-bb11-4252-b354-ac825b8a096f" containerID="6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1" exitCode=0 Oct 30 06:03:26.094243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.094093 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" event={"ID":"998663cb-bb11-4252-b354-ac825b8a096f","Type":"ContainerDied","Data":"6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1"} Oct 30 06:03:26.094243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.094167 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" event={"ID":"998663cb-bb11-4252-b354-ac825b8a096f","Type":"ContainerDied","Data":"e20c878079e18f43d1bdc0a68c792ee78f5d33f8bf16188c644d8cc4c4b23572"} Oct 30 06:03:26.094243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.094195 5980 scope.go:122] "RemoveContainer" containerID="6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1" Oct 30 06:03:26.094420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.094341 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr" Oct 30 06:03:26.096817 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.096788 5980 generic.go:356] "Generic (PLEG): container finished" podID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerID="b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1" exitCode=0 Oct 30 06:03:26.096944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.096884 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" Oct 30 06:03:26.096944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.096921 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" event={"ID":"a2d0fa5d-2434-476d-94dc-c209b4d180f2","Type":"ContainerDied","Data":"b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1"} Oct 30 06:03:26.097098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.097002 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz" event={"ID":"a2d0fa5d-2434-476d-94dc-c209b4d180f2","Type":"ContainerDied","Data":"2860670e9c21960b50f17977d019f35c52db25bb9f997555765753127d391e79"} Oct 30 06:03:26.111084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.111056 5980 scope.go:122] "RemoveContainer" containerID="6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1" Oct 30 06:03:26.111428 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:03:26.111401 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1\": container with ID starting with 6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1 not found: ID does not exist" containerID="6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1" Oct 30 06:03:26.111535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.111439 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1"} err="failed to get container status \"6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1\": rpc error: code = NotFound desc = could not find container \"6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1\": container with ID starting with 6dbd9b6e17ed526ebfb466dead82d15bd7487778ec3e36be070c5dae3e990ce1 not found: ID does not exist" Oct 30 06:03:26.111535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.111461 5980 scope.go:122] "RemoveContainer" containerID="b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1" Oct 30 06:03:26.124052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.124030 5980 scope.go:122] "RemoveContainer" containerID="b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1" Oct 30 06:03:26.124383 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:03:26.124360 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1\": container with ID starting with b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1 not found: ID does not exist" containerID="b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1" Oct 30 06:03:26.124477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.124393 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1"} err="failed to get container status \"b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1\": rpc error: code = NotFound desc = could not find container \"b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1\": container with ID starting with b04da1aa9ca94475b9a75dbd3e96b0c8d0e4a7d749a0c08b54307521aba161d1 not found: ID does not exist" Oct 30 06:03:26.133325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.133302 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx\" (UniqueName: \"kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx\") pod \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " Oct 30 06:03:26.133432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.133398 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume\") pod \"998663cb-bb11-4252-b354-ac825b8a096f\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " Oct 30 06:03:26.133524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.133508 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume\") pod \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\" (UID: \"a2d0fa5d-2434-476d-94dc-c209b4d180f2\") " Oct 30 06:03:26.133774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.133754 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r\" (UniqueName: \"kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r\") pod \"998663cb-bb11-4252-b354-ac825b8a096f\" (UID: \"998663cb-bb11-4252-b354-ac825b8a096f\") " Oct 30 06:03:26.133866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.133816 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume" pod "998663cb-bb11-4252-b354-ac825b8a096f" (UID: "998663cb-bb11-4252-b354-ac825b8a096f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:26.133923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.133864 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume" pod "a2d0fa5d-2434-476d-94dc-c209b4d180f2" (UID: "a2d0fa5d-2434-476d-94dc-c209b4d180f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:26.135491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.135469 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998663cb-bb11-4252-b354-ac825b8a096f-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:26.135610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.135494 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2d0fa5d-2434-476d-94dc-c209b4d180f2-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:26.135765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.135746 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx" pod "a2d0fa5d-2434-476d-94dc-c209b4d180f2" (UID: "a2d0fa5d-2434-476d-94dc-c209b4d180f2"). InnerVolumeSpecName "kube-api-access-6x8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:26.136569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.136549 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r" pod "998663cb-bb11-4252-b354-ac825b8a096f" (UID: "998663cb-bb11-4252-b354-ac825b8a096f"). InnerVolumeSpecName "kube-api-access-h2h4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:26.237046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.236965 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2h4r\" (UniqueName: \"kubernetes.io/projected/998663cb-bb11-4252-b354-ac825b8a096f-kube-api-access-h2h4r\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:26.237046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.236994 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8rx\" (UniqueName: \"kubernetes.io/projected/a2d0fa5d-2434-476d-94dc-c209b4d180f2-kube-api-access-6x8rx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:26.450117 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.450085 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz"] Oct 30 06:03:26.455144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.455096 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-862dc2-757889cb9f-p98bz"] Oct 30 06:03:26.495119 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.493128 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr"] Oct 30 06:03:26.501252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.501222 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-862dc2-6d787cc68c-zr5tr"] Oct 30 06:03:26.880340 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:26.880310 5980 scope.go:122] "RemoveContainer" containerID="6f4c7834647c50d87e0a4eee05e2f158c0fd41d2a569a6eac2a37b51e58a4d14" Oct 30 06:03:27.262161 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:27.262022 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-15.ec2.internal" podUID="eddf44eee240eccf9696b703c5918959" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:28.017257 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:28.017222 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998663cb-bb11-4252-b354-ac825b8a096f" path="/var/lib/kubelet/pods/998663cb-bb11-4252-b354-ac825b8a096f/volumes" Oct 30 06:03:28.017868 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:28.017849 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" path="/var/lib/kubelet/pods/a2d0fa5d-2434-476d-94dc-c209b4d180f2/volumes" Oct 30 06:03:35.399317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:35.399260 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:03:35.399777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:35.399339 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:37.278232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:37.277926 5980 generic.go:356] "Generic (PLEG): container finished" podID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerID="39ed0ad813770ba42c775c05df852a1c62e3a6e5bba9c4eef9fa1d0840daa8ad" exitCode=0 Oct 30 06:03:37.278232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:37.278160 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" event={"ID":"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f","Type":"ContainerDied","Data":"39ed0ad813770ba42c775c05df852a1c62e3a6e5bba9c4eef9fa1d0840daa8ad"} Oct 30 06:03:37.278737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:37.278693 5980 scope.go:122] "RemoveContainer" containerID="39ed0ad813770ba42c775c05df852a1c62e3a6e5bba9c4eef9fa1d0840daa8ad" Oct 30 06:03:37.284608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:37.284554 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:37.284745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:37.284655 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:37.284745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:37.284727 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:38.289229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:38.289188 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" event={"ID":"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f","Type":"ContainerStarted","Data":"03249d468a46d7b6c5ded63a4c167799916567f37b8eca6a7f873f4e36d6ac19"} Oct 30 06:03:38.843522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:38.843485 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:03:38.843800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:38.843772 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:03:38.843898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:38.843834 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 06:03:39.844152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:39.844102 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:03:39.844583 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:39.844178 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 06:03:40.843780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:40.843742 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:03:40.844042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:40.843807 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 06:03:41.844335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:41.844300 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:03:41.844767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:41.844356 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 06:03:42.843757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:42.843724 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:03:42.843972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:42.843777 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 06:03:43.122388 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:03:43.122302 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:03:43.844507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:43.844467 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 06:03:43.844691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:43.844553 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 06:03:44.848220 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:44.848191 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:03:44.848688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:44.848672 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:03:44.849308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:44.849290 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:03:45.606273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:45.606238 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs"] Oct 30 06:03:45.606650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:45.606625 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" containerID="cri-o://0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7" gracePeriod=30 Oct 30 06:03:47.691656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:47.691616 5980 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:03:47.692078 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:47.691701 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:03:50.902888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.902862 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:03:50.972458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.972427 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.972820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.972777 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:50.972933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.972860 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.972933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.972914 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.973043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.972993 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.973163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.973041 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.973163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.973097 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.973163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.973141 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.973327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.973247 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.973386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.973367 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft\") pod \"91aa81c6-2469-4e68-abed-712ffa461dd8\" (UID: \"91aa81c6-2469-4e68-abed-712ffa461dd8\") " Oct 30 06:03:50.974011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.973992 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:50.975557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.974729 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:50.975890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.975175 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:50.976008 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.975226 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:50.976322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.975455 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:50.978199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.978161 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "kube-api-access-6kvft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:50.979469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.979430 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:50.981182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.981099 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:03:50.981182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:50.981170 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token" pod "91aa81c6-2469-4e68-abed-712ffa461dd8" (UID: "91aa81c6-2469-4e68-abed-712ffa461dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:51.079386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.079348 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvft\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-kube-api-access-6kvft\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.079612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.079595 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.079707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.079696 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aa81c6-2469-4e68-abed-712ffa461dd8-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.079790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.079780 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.079872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.079862 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.079954 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.079944 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.080028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.080019 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.080125 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.080097 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aa81c6-2469-4e68-abed-712ffa461dd8-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.421890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.421838 5980 generic.go:356] "Generic (PLEG): container finished" podID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerID="0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7" exitCode=0 Oct 30 06:03:51.422097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.421925 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" Oct 30 06:03:51.422097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.421924 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" event={"ID":"91aa81c6-2469-4e68-abed-712ffa461dd8","Type":"ContainerDied","Data":"0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7"} Oct 30 06:03:51.422097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.422077 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs" event={"ID":"91aa81c6-2469-4e68-abed-712ffa461dd8","Type":"ContainerDied","Data":"4e5528257e409b3e4d95141f4d15f72a0bb0cf283c6b71ada7febc2b2cdebddc"} Oct 30 06:03:51.422271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.422101 5980 scope.go:122] "RemoveContainer" containerID="0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7" Oct 30 06:03:51.439596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.439576 5980 scope.go:122] "RemoveContainer" containerID="0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7" Oct 30 06:03:51.439895 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:03:51.439876 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7\": container with ID starting with 0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7 not found: ID does not exist" containerID="0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7" Oct 30 06:03:51.439940 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.439907 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7"} err="failed to get container status \"0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7\": rpc error: code = NotFound desc = could not find container \"0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7\": container with ID starting with 0e01cd452423d32b2a98c034d484f52077197789821b6e4e8dbe4dedb73b5ef7 not found: ID does not exist" Oct 30 06:03:51.479748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.479718 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs"] Oct 30 06:03:51.493201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:51.493171 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-vl4zs"] Oct 30 06:03:52.016787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:52.016739 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" path="/var/lib/kubelet/pods/91aa81c6-2469-4e68-abed-712ffa461dd8/volumes" Oct 30 06:03:56.534573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.534536 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg"] Oct 30 06:03:56.535502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535479 5980 cpu_manager.go:414] "containerMap: removing container" podUID="998663cb-bb11-4252-b354-ac825b8a096f" containerName="payload-pre-processing" Oct 30 06:03:56.535555 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535506 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998663cb-bb11-4252-b354-ac825b8a096f" containerName="payload-pre-processing" Oct 30 06:03:56.535596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535558 5980 cpu_manager.go:414] "containerMap: removing container" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" Oct 30 06:03:56.535596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535566 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" Oct 30 06:03:56.535664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535616 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerName="payload-processing" Oct 30 06:03:56.535664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535623 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerName="payload-processing" Oct 30 06:03:56.535843 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535828 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="998663cb-bb11-4252-b354-ac825b8a096f" containerName="payload-pre-processing" Oct 30 06:03:56.535876 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535859 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aa81c6-2469-4e68-abed-712ffa461dd8" containerName="istio-proxy" Oct 30 06:03:56.535908 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.535877 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d0fa5d-2434-476d-94dc-c209b4d180f2" containerName="payload-processing" Oct 30 06:03:56.540228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.540203 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.561406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.561363 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg"] Oct 30 06:03:56.569140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.569080 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-387b16\"/\"test-model-387b16-kserve-self-signed-certs\"" Oct 30 06:03:56.569140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.569123 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-387b16\"/\"default-dockercfg-zvhs4\"" Oct 30 06:03:56.570233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.570215 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-387b16\"/\"openshift-service-ca.crt\"" Oct 30 06:03:56.570480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.570465 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-387b16\"/\"kube-root-ca.crt\"" Oct 30 06:03:56.687807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.687771 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.688001 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.687980 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.688065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.688047 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.688198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.688182 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.688244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.688219 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.688288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.688249 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jknlq\" (UniqueName: \"kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.801089 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.801184 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.801234 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jknlq\" (UniqueName: \"kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.801401 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.801547 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.801608 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.802168 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.802996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.802964 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.809131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.803407 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.850132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.840436 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jknlq\" (UniqueName: \"kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.883134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.882917 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:56.883411 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:56.883215 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm\") pod \"test-model-387b16-kserve-68b5765597-c2qtg\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:57.171133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:57.170288 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:03:57.409231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:57.409199 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j"] Oct 30 06:03:57.531726 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:03:57.531695 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76fa82d8_d669_4aeb_acab_6f9714712db5.slice/crio-2b216c88f3cfe5acad4e0543e3bc70e3cc5585956a9f0c90289501155a42e050 WatchSource:0}: Error finding container 2b216c88f3cfe5acad4e0543e3bc70e3cc5585956a9f0c90289501155a42e050: Status 404 returned error can't find the container with id 2b216c88f3cfe5acad4e0543e3bc70e3cc5585956a9f0c90289501155a42e050 Oct 30 06:03:57.531898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:57.531722 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg"] Oct 30 06:03:58.484092 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:58.484057 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" event={"ID":"76fa82d8-d669-4aeb-acab-6f9714712db5","Type":"ContainerStarted","Data":"4116ff2498c08d51c6cd10390b9473f2706cf9dee9f9a4d20f8bb07cff93e331"} Oct 30 06:03:58.484092 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:58.484117 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" event={"ID":"76fa82d8-d669-4aeb-acab-6f9714712db5","Type":"ContainerStarted","Data":"2b216c88f3cfe5acad4e0543e3bc70e3cc5585956a9f0c90289501155a42e050"} Oct 30 06:03:58.529808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:58.529759 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" podStartSLOduration=2.529744053 podStartE2EDuration="2.529744053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:58.528152117 +0000 UTC m=+1892.636191156" watchObservedRunningTime="2026-10-30 06:03:58.529744053 +0000 UTC m=+1892.637783085" Oct 30 06:03:59.176517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:03:59.176477 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:04:02.410557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:02.410474 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" containerName="payload-pre-processing" containerID="cri-o://85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d" gracePeriod=25 Oct 30 06:04:07.176926 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:07.176515 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:04:07.215655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:07.215631 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:04:07.398644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:07.398086 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:04:07.839129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:07.831223 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:04:13.090433 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:04:13.090396 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:04:17.391858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:17.391817 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:04:22.733850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.733817 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 06:04:22.839650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.839616 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8\" (UniqueName: \"kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8\") pod \"89a73015-84a1-475b-9376-d5a149f94ae5\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " Oct 30 06:04:22.839918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.839899 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume\") pod \"89a73015-84a1-475b-9376-d5a149f94ae5\" (UID: \"89a73015-84a1-475b-9376-d5a149f94ae5\") " Oct 30 06:04:22.840442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.840371 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume" pod "89a73015-84a1-475b-9376-d5a149f94ae5" (UID: "89a73015-84a1-475b-9376-d5a149f94ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:22.840651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.840636 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a73015-84a1-475b-9376-d5a149f94ae5-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:22.842215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.842173 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8" pod "89a73015-84a1-475b-9376-d5a149f94ae5" (UID: "89a73015-84a1-475b-9376-d5a149f94ae5"). InnerVolumeSpecName "kube-api-access-pqbn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:22.871589 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.871552 5980 generic.go:356] "Generic (PLEG): container finished" podID="89a73015-84a1-475b-9376-d5a149f94ae5" containerID="85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d" exitCode=0 Oct 30 06:04:22.871838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.871613 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" Oct 30 06:04:22.871838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.871762 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" event={"ID":"89a73015-84a1-475b-9376-d5a149f94ae5","Type":"ContainerDied","Data":"85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d"} Oct 30 06:04:22.871974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.871847 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j" event={"ID":"89a73015-84a1-475b-9376-d5a149f94ae5","Type":"ContainerDied","Data":"d6b502d373938818cad57e66dee6411f8ea1eeaf5bc1c93e063c8bfb6f3421a2"} Oct 30 06:04:22.871974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.871873 5980 scope.go:122] "RemoveContainer" containerID="85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d" Oct 30 06:04:22.887702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.887659 5980 scope.go:122] "RemoveContainer" containerID="85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d" Oct 30 06:04:22.888174 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:04:22.888042 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d\": container with ID starting with 85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d not found: ID does not exist" containerID="85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d" Oct 30 06:04:22.888174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.888073 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d"} err="failed to get container status \"85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d\": rpc error: code = NotFound desc = could not find container \"85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d\": container with ID starting with 85a0469054f6cc16c109f608211734818cf3bdc9137306cc105ffa530c13167d not found: ID does not exist" Oct 30 06:04:22.942752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.942669 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqbn8\" (UniqueName: \"kubernetes.io/projected/89a73015-84a1-475b-9376-d5a149f94ae5-kube-api-access-pqbn8\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:22.947178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.947150 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j"] Oct 30 06:04:22.952084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:22.952048 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-de7242-5f4bd94c-tsd2j"] Oct 30 06:04:23.812160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:23.812121 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w"] Oct 30 06:04:23.812660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:23.812398 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" containerID="cri-o://7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939" gracePeriod=30 Oct 30 06:04:24.015288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:24.015253 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" path="/var/lib/kubelet/pods/89a73015-84a1-475b-9376-d5a149f94ae5/volumes" Oct 30 06:04:29.114349 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.114316 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 06:04:29.255915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.255865 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.255915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.255922 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256006 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256085 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256178 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256251 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:29.256386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256268 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256319 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:29.256386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256362 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256550 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256663 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert\") pod \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\" (UID: \"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c\") " Oct 30 06:04:29.256920 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256688 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:29.256993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.256711 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:29.257250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.257147 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:29.257959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.257939 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.258071 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.258060 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.258180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.258168 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.258270 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.258258 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.258347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.258336 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.258906 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.258880 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:29.259380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.259348 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "kube-api-access-f8b5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:29.259471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.259447 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:04:29.259471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.259446 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token" pod "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" (UID: "9e0172c7-17db-49ea-88ec-cff7ad1e9a6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:29.360016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.359934 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.360016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.359966 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8b5c\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-kube-api-access-f8b5c\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.360016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.359976 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.360016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.359986 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:29.931704 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.931664 5980 generic.go:356] "Generic (PLEG): container finished" podID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerID="7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939" exitCode=0 Oct 30 06:04:29.931916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.931730 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" Oct 30 06:04:29.931916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.931749 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" event={"ID":"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c","Type":"ContainerDied","Data":"7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939"} Oct 30 06:04:29.931916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.931834 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w" event={"ID":"9e0172c7-17db-49ea-88ec-cff7ad1e9a6c","Type":"ContainerDied","Data":"dcd995a750dac19cddef3dfb430256cd94bcf694e32fbd1a2acaf371ca916479"} Oct 30 06:04:29.931916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.931863 5980 scope.go:122] "RemoveContainer" containerID="7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939" Oct 30 06:04:29.944758 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.944739 5980 scope.go:122] "RemoveContainer" containerID="7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939" Oct 30 06:04:29.945145 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:04:29.945124 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939\": container with ID starting with 7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939 not found: ID does not exist" containerID="7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939" Oct 30 06:04:29.945225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:29.945155 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939"} err="failed to get container status \"7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939\": rpc error: code = NotFound desc = could not find container \"7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939\": container with ID starting with 7e987fca1a38c7770cfdeaa9c57c91d176234f12b0023a1d97b5e8bed5843939 not found: ID does not exist" Oct 30 06:04:30.016738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:30.016707 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w"] Oct 30 06:04:30.016897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:30.016748 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-2m66w"] Oct 30 06:04:32.016403 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:32.016369 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" path="/var/lib/kubelet/pods/9e0172c7-17db-49ea-88ec-cff7ad1e9a6c/volumes" Oct 30 06:04:37.048520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.048484 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt"] Oct 30 06:04:37.049457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.049438 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" Oct 30 06:04:37.049457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.049458 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" Oct 30 06:04:37.049604 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.049499 5980 cpu_manager.go:414] "containerMap: removing container" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" containerName="payload-pre-processing" Oct 30 06:04:37.049604 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.049505 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" containerName="payload-pre-processing" Oct 30 06:04:37.049708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.049621 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0172c7-17db-49ea-88ec-cff7ad1e9a6c" containerName="istio-proxy" Oct 30 06:04:37.049708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.049636 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a73015-84a1-475b-9376-d5a149f94ae5" containerName="payload-pre-processing" Oct 30 06:04:37.054241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.054220 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.070627 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.070586 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aee070-openshift-default-dockercfg-stq9x\"" Oct 30 06:04:37.090550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.090513 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt"] Oct 30 06:04:37.120772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.120731 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.120972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.120810 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.120972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.120921 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh8xs\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.120972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.120950 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.121198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.121024 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.121198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.121097 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.121198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.121167 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.121366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.121245 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.121366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.121360 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223338 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223435 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223475 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223539 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh8xs\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223559 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223795 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223632 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223795 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223709 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223795 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223762 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.223963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.223845 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.224668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.224455 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.224668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.224562 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.224668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.224575 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.224668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.224597 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.224983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.224704 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.226380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.226350 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.226546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.226524 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.249043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.249009 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.249215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.249069 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh8xs\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-855wt\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.370712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.370656 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:37.707862 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:04:37.707822 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d66026_513e_4a09_bf34_e11a07bec5aa.slice/crio-ad0c71f6fe97c9d32e92b4d1caab098098372971275552cd2d395b0e2ebde0df WatchSource:0}: Error finding container ad0c71f6fe97c9d32e92b4d1caab098098372971275552cd2d395b0e2ebde0df: Status 404 returned error can't find the container with id ad0c71f6fe97c9d32e92b4d1caab098098372971275552cd2d395b0e2ebde0df Oct 30 06:04:37.708574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.708548 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt"] Oct 30 06:04:37.710993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.710957 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:37.711080 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.711027 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:37.711080 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:37.711056 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:38.042356 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:38.042258 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" event={"ID":"e5d66026-513e-4a09-bf34-e11a07bec5aa","Type":"ContainerStarted","Data":"86b685673b0623af42f2f3fdedd192f7c73cdfa9634f2c589c322b87c09b432a"} Oct 30 06:04:38.042356 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:38.042321 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" event={"ID":"e5d66026-513e-4a09-bf34-e11a07bec5aa","Type":"ContainerStarted","Data":"ad0c71f6fe97c9d32e92b4d1caab098098372971275552cd2d395b0e2ebde0df"} Oct 30 06:04:38.091736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:38.091680 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podStartSLOduration=1.091665027 podStartE2EDuration="1.091665027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:38.089959154 +0000 UTC m=+1932.197998188" watchObservedRunningTime="2026-10-30 06:04:38.091665027 +0000 UTC m=+1932.199704059" Oct 30 06:04:38.373040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:38.371304 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:38.376034 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:38.375995 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:38.376299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:38.376260 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:39.385643 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:39.385606 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:39.393335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:39.393242 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:39.715226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:39.711373 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2"] Oct 30 06:04:39.715226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:39.711674 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" containerID="cri-o://54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c" gracePeriod=30 Oct 30 06:04:40.342695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:40.342656 5980 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:04:40.342882 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:40.342743 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:04:40.385337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:40.384537 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:40.385337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:40.384606 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:41.029520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.027020 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr"] Oct 30 06:04:41.062142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.060500 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.099132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.083570 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr"] Oct 30 06:04:41.199156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.198328 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cfe14f-openshift-default-dockercfg-hbfwm\"" Oct 30 06:04:41.280857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.280728 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288301 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288359 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288406 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288461 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288508 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6b8\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288542 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288752 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.291178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.288861 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.373306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.373273 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:41.373570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.373547 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:41.399058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.398940 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.408194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.408160 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.408728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.408698 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.413153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.408953 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.413422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.413392 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.413933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.413914 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.414516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.414496 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.428775 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.428722 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6b8\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.429082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.429048 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.431937 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.402670 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.432138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.429266 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.432272 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.413836 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.432400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.409502 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.432512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.430336 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.447336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.438369 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.447336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.446841 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.449643 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.449489 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.455219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.455162 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6b8\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:41.612179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:41.612145 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:42.432821 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:42.428349 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:42.432821 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:42.428435 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:42.439079 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:42.439039 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr"] Oct 30 06:04:42.507935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:42.507884 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:42.508184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:42.508005 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:42.508339 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:42.508300 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:43.129230 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:04:43.129189 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:04:43.385203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.384613 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:43.385203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.384689 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:43.474143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.474081 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" event={"ID":"c53c62e9-60a6-44d4-94d1-9391b1b3a14a","Type":"ContainerStarted","Data":"20460ffbf18a89958ceb88fb219f9982c443c5f492e794b51515649c1eebf3c4"} Oct 30 06:04:43.474765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.474736 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" event={"ID":"c53c62e9-60a6-44d4-94d1-9391b1b3a14a","Type":"ContainerStarted","Data":"ac78945b5328240a21a76d8190067053392b6a21209cc1b84dc49581440dd996"} Oct 30 06:04:43.608788 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.608709 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podStartSLOduration=3.608690426 podStartE2EDuration="3.608690426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:43.59231198 +0000 UTC m=+1937.700351016" watchObservedRunningTime="2026-10-30 06:04:43.608690426 +0000 UTC m=+1937.716729460" Oct 30 06:04:43.612741 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.612709 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:43.619480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:43.619452 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:44.386135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:44.384485 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:44.386135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:44.384545 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:44.503127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:44.503084 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:44.505050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:44.505014 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:04:45.090494 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.090466 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:04:45.160997 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.160953 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161019 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161166 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161246 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161292 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.161490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161417 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161483 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161635 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161559 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161631 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.161712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.161663 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket\") pod \"14526aed-4138-40db-96a4-ffac615cfebd\" (UID: \"14526aed-4138-40db-96a4-ffac615cfebd\") " Oct 30 06:04:45.162565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.162537 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.162710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.162684 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.163291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.163227 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.163401 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.163384 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:45.163469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.163439 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.177523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.173261 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.177523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.173322 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:45.190410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.190339 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "kube-api-access-7mclj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:45.193790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.193742 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo" pod "14526aed-4138-40db-96a4-ffac615cfebd" (UID: "14526aed-4138-40db-96a4-ffac615cfebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:04:45.264808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264755 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.264808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264804 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.264808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264820 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.265187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264833 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14526aed-4138-40db-96a4-ffac615cfebd-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.265187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264847 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14526aed-4138-40db-96a4-ffac615cfebd-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.265187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264859 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.265187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264872 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mclj\" (UniqueName: \"kubernetes.io/projected/14526aed-4138-40db-96a4-ffac615cfebd-kube-api-access-7mclj\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.265187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.264884 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14526aed-4138-40db-96a4-ffac615cfebd-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.371766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.371722 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 06:04:45.371973 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.371798 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 06:04:45.515693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.515657 5980 generic.go:356] "Generic (PLEG): container finished" podID="14526aed-4138-40db-96a4-ffac615cfebd" containerID="54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c" exitCode=0 Oct 30 06:04:45.516185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.515748 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" event={"ID":"14526aed-4138-40db-96a4-ffac615cfebd","Type":"ContainerDied","Data":"54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c"} Oct 30 06:04:45.516185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.515817 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" event={"ID":"14526aed-4138-40db-96a4-ffac615cfebd","Type":"ContainerDied","Data":"3e52c08b8d357bc7c90a49d55135fa0f7274e504a07056139dba26fc1fea38af"} Oct 30 06:04:45.516185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.515723 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2" Oct 30 06:04:45.516185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.515855 5980 scope.go:122] "RemoveContainer" containerID="54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c" Oct 30 06:04:45.529865 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.529841 5980 scope.go:122] "RemoveContainer" containerID="54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c" Oct 30 06:04:45.530229 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:04:45.530204 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c\": container with ID starting with 54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c not found: ID does not exist" containerID="54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c" Oct 30 06:04:45.530318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.530241 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c"} err="failed to get container status \"54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c\": rpc error: code = NotFound desc = could not find container \"54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c\": container with ID starting with 54c37285683d32e6f19abadf0fcf6be584b09a7a91e37b532b0590875694f00c not found: ID does not exist" Oct 30 06:04:45.578040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.577550 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2"] Oct 30 06:04:45.594521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:45.594487 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-kf8b2"] Oct 30 06:04:46.033879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:46.033847 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14526aed-4138-40db-96a4-ffac615cfebd" path="/var/lib/kubelet/pods/14526aed-4138-40db-96a4-ffac615cfebd/volumes" Oct 30 06:04:46.377638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:46.377607 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:46.378017 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:46.377993 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:46.378748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:46.378731 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:04:47.790080 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.790041 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj"] Oct 30 06:04:47.790849 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.790833 5980 cpu_manager.go:414] "containerMap: removing container" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" Oct 30 06:04:47.790897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.790851 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" Oct 30 06:04:47.791003 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.790993 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="14526aed-4138-40db-96a4-ffac615cfebd" containerName="istio-proxy" Oct 30 06:04:47.794537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.794519 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:47.814289 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.814242 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj"] Oct 30 06:04:47.822964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.822933 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cfe14f\"" Oct 30 06:04:47.823432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.823371 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-cfe14f-dockercfg-rwmk6\"" Oct 30 06:04:48.003135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:47.976143 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.159403 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.155964 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgjw\" (UniqueName: \"kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw\") pod \"payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.177798 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.177730 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm"] Oct 30 06:04:48.197833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.197780 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm"] Oct 30 06:04:48.198085 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.197894 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.262219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.262183 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.262446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.262358 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.262535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.262519 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgjw\" (UniqueName: \"kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw\") pod \"payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.262736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.262712 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rfsc\" (UniqueName: \"kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc\") pod \"payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.263794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.263768 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.295261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.295224 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgjw\" (UniqueName: \"kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw\") pod \"payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.364736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.364700 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rfsc\" (UniqueName: \"kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc\") pod \"payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.364921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.364906 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.365653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.365633 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.391088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.391051 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rfsc\" (UniqueName: \"kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc\") pod \"payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.416465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.416379 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:48.523845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.523610 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:48.740880 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:04:48.740844 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6d5408_690c_4535_9740_64d45c107c05.slice/crio-778c28ac04ccb28cbb4d3b06cfb483983dcacd60113ddda9d533a445fd536d08 WatchSource:0}: Error finding container 778c28ac04ccb28cbb4d3b06cfb483983dcacd60113ddda9d533a445fd536d08: Status 404 returned error can't find the container with id 778c28ac04ccb28cbb4d3b06cfb483983dcacd60113ddda9d533a445fd536d08 Oct 30 06:04:48.742767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.742739 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj"] Oct 30 06:04:48.856816 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:48.856778 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm"] Oct 30 06:04:48.857246 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:04:48.856857 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5301a71a_b051_42fd_a14e_2ae51a411dd5.slice/crio-9e49f3e91cb4919680a625e4a56d85618083623a9d172250fcd5da3669c79195 WatchSource:0}: Error finding container 9e49f3e91cb4919680a625e4a56d85618083623a9d172250fcd5da3669c79195: Status 404 returned error can't find the container with id 9e49f3e91cb4919680a625e4a56d85618083623a9d172250fcd5da3669c79195 Oct 30 06:04:49.600516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.599071 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" event={"ID":"ba6d5408-690c-4535-9740-64d45c107c05","Type":"ContainerStarted","Data":"12e6a8948ffebe796189004c3f041f0b94f48f0dea34a53c6fdb4b001d8fc948"} Oct 30 06:04:49.600516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.599208 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" event={"ID":"ba6d5408-690c-4535-9740-64d45c107c05","Type":"ContainerStarted","Data":"778c28ac04ccb28cbb4d3b06cfb483983dcacd60113ddda9d533a445fd536d08"} Oct 30 06:04:49.600516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.600464 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:04:49.628455 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.628408 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" event={"ID":"5301a71a-b051-42fd-a14e-2ae51a411dd5","Type":"ContainerStarted","Data":"5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24"} Oct 30 06:04:49.628660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.628493 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" event={"ID":"5301a71a-b051-42fd-a14e-2ae51a411dd5","Type":"ContainerStarted","Data":"9e49f3e91cb4919680a625e4a56d85618083623a9d172250fcd5da3669c79195"} Oct 30 06:04:49.628660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.628644 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:04:49.655834 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.655783 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" podStartSLOduration=2.65576831 podStartE2EDuration="2.65576831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:49.652518729 +0000 UTC m=+1943.760557761" watchObservedRunningTime="2026-10-30 06:04:49.65576831 +0000 UTC m=+1943.763807342" Oct 30 06:04:49.698410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:49.698339 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" podStartSLOduration=2.698325122 podStartE2EDuration="2.698325122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:49.697440493 +0000 UTC m=+1943.805479527" watchObservedRunningTime="2026-10-30 06:04:49.698325122 +0000 UTC m=+1943.806364154" Oct 30 06:04:58.602208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.591184 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8"] Oct 30 06:04:58.674146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.655640 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:58.701160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.691594 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d3e7e6\"" Oct 30 06:04:58.715133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.713299 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdkh\" (UniqueName: \"kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh\") pod \"payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:58.715133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.713550 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:58.725378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.722056 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8"] Oct 30 06:04:58.787564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.774524 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d3e7e6-dockercfg-wbbtg\"" Oct 30 06:04:58.817219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.815587 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdkh\" (UniqueName: \"kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh\") pod \"payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:58.817219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.815783 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:58.817219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.816859 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:58.941228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:58.939536 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdkh\" (UniqueName: \"kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh\") pod \"payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:59.109474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:59.109439 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:04:59.443088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:59.443057 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8"] Oct 30 06:04:59.443328 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:04:59.443250 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06b7249_21eb_435a_aa4d_3c6b5c6e4014.slice/crio-70ad9697be3936e1c658d270c92eafc097248d31bdc22c89c43c76ef011b23b7 WatchSource:0}: Error finding container 70ad9697be3936e1c658d270c92eafc097248d31bdc22c89c43c76ef011b23b7: Status 404 returned error can't find the container with id 70ad9697be3936e1c658d270c92eafc097248d31bdc22c89c43c76ef011b23b7 Oct 30 06:04:59.813630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:59.813546 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" event={"ID":"d06b7249-21eb-435a-aa4d-3c6b5c6e4014","Type":"ContainerStarted","Data":"c4127e08ad0c4ffc0721bd35f266547e0869fd2581577f098331694d2c23b374"} Oct 30 06:04:59.813630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:59.813603 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" event={"ID":"d06b7249-21eb-435a-aa4d-3c6b5c6e4014","Type":"ContainerStarted","Data":"70ad9697be3936e1c658d270c92eafc097248d31bdc22c89c43c76ef011b23b7"} Oct 30 06:04:59.814066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:04:59.813654 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:05:00.646182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:00.645132 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:05:00.706436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:00.706376 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" podStartSLOduration=2.706353144 podStartE2EDuration="2.706353144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:59.876884432 +0000 UTC m=+1953.984923490" watchObservedRunningTime="2026-10-30 06:05:00.706353144 +0000 UTC m=+1954.814392183" Oct 30 06:05:01.648322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:01.648288 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:05:10.851783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:10.851754 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:05:13.090240 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:05:13.090196 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:05:28.578779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.578743 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj"] Oct 30 06:05:28.584424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.584405 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.600671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.600645 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d64c9\"/\"kube-root-ca.crt\"" Oct 30 06:05:28.600671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.600645 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d64c9\"/\"openshift-service-ca.crt\"" Oct 30 06:05:28.618429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.618346 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d64c9\"/\"default-dockercfg-psrlq\"" Oct 30 06:05:28.618429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.618363 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d64c9\"/\"test-model-0d64c9-kserve-self-signed-certs\"" Oct 30 06:05:28.620824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.620782 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj"] Oct 30 06:05:28.700490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.700453 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.700665 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.700520 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnqz\" (UniqueName: \"kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.700710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.700679 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.700762 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.700729 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.700849 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.700829 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.700973 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.700959 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.802893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.802859 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.802978 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803015 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnqz\" (UniqueName: \"kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803154 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803096 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803154 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803208 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803267 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803463 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.803626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.803590 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.805662 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.805637 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.808671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.808647 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.829397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.829338 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnqz\" (UniqueName: \"kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz\") pod \"test-model-0d64c9-kserve-c747557bc-zdhlj\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:28.899081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:28.899051 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:29.312919 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:05:29.312887 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ec9e425_dacf_4638_aa69_7abfff7f0aa0.slice/crio-6c66879ba66c5253d64550e65b8dbf9b4433a76ac3c698881f0c922089e50139 WatchSource:0}: Error finding container 6c66879ba66c5253d64550e65b8dbf9b4433a76ac3c698881f0c922089e50139: Status 404 returned error can't find the container with id 6c66879ba66c5253d64550e65b8dbf9b4433a76ac3c698881f0c922089e50139 Oct 30 06:05:29.314527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:29.314501 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj"] Oct 30 06:05:30.145419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:30.145285 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" event={"ID":"7ec9e425-dacf-4638-aa69-7abfff7f0aa0","Type":"ContainerStarted","Data":"2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b"} Oct 30 06:05:30.145419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:30.145368 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" event={"ID":"7ec9e425-dacf-4638-aa69-7abfff7f0aa0","Type":"ContainerStarted","Data":"6c66879ba66c5253d64550e65b8dbf9b4433a76ac3c698881f0c922089e50139"} Oct 30 06:05:30.193232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:30.193169 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" podStartSLOduration=2.1931544609999998 podStartE2EDuration="2.193154461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:30.188724685 +0000 UTC m=+1984.296763718" watchObservedRunningTime="2026-10-30 06:05:30.193154461 +0000 UTC m=+1984.301193493" Oct 30 06:05:30.899817 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:30.899780 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:38.899565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:38.899481 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:38.915933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:38.915904 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:39.032432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.032395 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh"] Oct 30 06:05:39.032701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.032673 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" containerID="cri-o://2cff76b764d09b394473cac8665d275dabecbe579539474fc133525da4c54fcb" gracePeriod=30 Oct 30 06:05:39.036475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.036448 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x"] Oct 30 06:05:39.036712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.036666 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" containerID="cri-o://3d9531426d233f95eebf51ce413a51bac8832676b99d67568f4ce4d66ebaaed9" gracePeriod=30 Oct 30 06:05:39.234557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.234465 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:05:39.788973 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.788934 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7"] Oct 30 06:05:39.789219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:39.789152 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" containerID="cri-o://1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc" gracePeriod=30 Oct 30 06:05:43.091084 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:05:43.091046 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:05:43.560898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:43.560795 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:05:43.560898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:43.560873 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:05:44.272284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.272248 5980 generic.go:356] "Generic (PLEG): container finished" podID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerID="3d9531426d233f95eebf51ce413a51bac8832676b99d67568f4ce4d66ebaaed9" exitCode=0 Oct 30 06:05:44.272838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.272491 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" event={"ID":"61eb1e21-74fd-4bdf-9782-fa4a081782e3","Type":"ContainerDied","Data":"3d9531426d233f95eebf51ce413a51bac8832676b99d67568f4ce4d66ebaaed9"} Oct 30 06:05:44.276623 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.276231 5980 generic.go:356] "Generic (PLEG): container finished" podID="b793ae0f-d086-4c38-af08-d68376033083" containerID="2cff76b764d09b394473cac8665d275dabecbe579539474fc133525da4c54fcb" exitCode=0 Oct 30 06:05:44.276623 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.276261 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" event={"ID":"b793ae0f-d086-4c38-af08-d68376033083","Type":"ContainerDied","Data":"2cff76b764d09b394473cac8665d275dabecbe579539474fc133525da4c54fcb"} Oct 30 06:05:44.381292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.380783 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 06:05:44.407914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.407888 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 06:05:44.450926 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.450891 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451001 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451054 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451088 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451236 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451229 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.451379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451302 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451373 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451552 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451430 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.451552 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451441 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.451552 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451471 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:44.451691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451616 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy\") pod \"b793ae0f-d086-4c38-af08-d68376033083\" (UID: \"b793ae0f-d086-4c38-af08-d68376033083\") " Oct 30 06:05:44.452073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451807 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.452073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.451822 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.452717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.452669 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.452717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.452700 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.452717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.452714 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.452943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.452730 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.452943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.452746 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b793ae0f-d086-4c38-af08-d68376033083-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.453946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.453916 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "kube-api-access-xdwn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:44.454351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.454330 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:44.454423 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.454384 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:44.454423 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.454386 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy" pod "b793ae0f-d086-4c38-af08-d68376033083" (UID: "b793ae0f-d086-4c38-af08-d68376033083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.553607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553572 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.553812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553631 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.553812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553669 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.553812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553729 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.553978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553859 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.553978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553892 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.553978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553936 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.554282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.553960 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.554282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.554004 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.554282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.554093 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.554608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.554589 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert\") pod \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\" (UID: \"61eb1e21-74fd-4bdf-9782-fa4a081782e3\") " Oct 30 06:05:44.554695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.554626 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.554695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.554651 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.555038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.554981 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555526 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555548 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555561 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwn8\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-kube-api-access-xdwn8\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555573 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b793ae0f-d086-4c38-af08-d68376033083-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555584 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b793ae0f-d086-4c38-af08-d68376033083-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555594 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555606 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555616 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.555655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.555627 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b793ae0f-d086-4c38-af08-d68376033083-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.557352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.557324 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "kube-api-access-fphbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:44.557457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.557401 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:44.558052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.558028 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:44.566313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.566276 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy" pod "61eb1e21-74fd-4bdf-9782-fa4a081782e3" (UID: "61eb1e21-74fd-4bdf-9782-fa4a081782e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:44.657706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.657608 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fphbz\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-kube-api-access-fphbz\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.657706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.657638 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.657706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.657651 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:44.657706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:44.657659 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61eb1e21-74fd-4bdf-9782-fa4a081782e3-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.112087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.112056 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 06:05:45.269661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269560 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.269661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269621 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.269661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269650 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.269966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269741 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.269966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269780 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.269966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269853 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.270143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.269978 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:45.270143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270017 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.270143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270079 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.270304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270194 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:45.270304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270204 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token\") pod \"afd53c91-9a4c-4243-a657-ce77ddd88282\" (UID: \"afd53c91-9a4c-4243-a657-ce77ddd88282\") " Oct 30 06:05:45.270432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270410 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:45.270484 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270442 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:45.270834 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.270801 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:45.271310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.271283 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.271424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.271314 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.271424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.271330 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.271424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.271343 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd53c91-9a4c-4243-a657-ce77ddd88282-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.271424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.271356 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.272996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.272964 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:45.273863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.273816 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:45.274082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.274056 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:45.274082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.274068 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4" pod "afd53c91-9a4c-4243-a657-ce77ddd88282" (UID: "afd53c91-9a4c-4243-a657-ce77ddd88282"). InnerVolumeSpecName "kube-api-access-x7sf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:45.310535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.310491 5980 generic.go:356] "Generic (PLEG): container finished" podID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerID="1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc" exitCode=0 Oct 30 06:05:45.310748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.310547 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" Oct 30 06:05:45.310748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.310576 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" event={"ID":"afd53c91-9a4c-4243-a657-ce77ddd88282","Type":"ContainerDied","Data":"1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc"} Oct 30 06:05:45.310748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.310644 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7" event={"ID":"afd53c91-9a4c-4243-a657-ce77ddd88282","Type":"ContainerDied","Data":"cbd7810fb27e8d12d9aa7f978e60bb911158b7dd3e53289b01497875c5fcc3b3"} Oct 30 06:05:45.310748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.310659 5980 scope.go:122] "RemoveContainer" containerID="1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc" Oct 30 06:05:45.313887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.313859 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" Oct 30 06:05:45.314656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.314206 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh" event={"ID":"b793ae0f-d086-4c38-af08-d68376033083","Type":"ContainerDied","Data":"31e709103e413d005a8b363f7846a08b90183fd4c94f6cd699d7b33ce5ea9c54"} Oct 30 06:05:45.316360 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.316328 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" event={"ID":"61eb1e21-74fd-4bdf-9782-fa4a081782e3","Type":"ContainerDied","Data":"71ff45bf94488b1536745995364ddc051621be2dc7d4f380355c0bde31377929"} Oct 30 06:05:45.316474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.316448 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x" Oct 30 06:05:45.331653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.331627 5980 scope.go:122] "RemoveContainer" containerID="1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc" Oct 30 06:05:45.332272 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:05:45.332239 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc\": container with ID starting with 1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc not found: ID does not exist" containerID="1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc" Oct 30 06:05:45.332398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.332294 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc"} err="failed to get container status \"1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc\": rpc error: code = NotFound desc = could not find container \"1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc\": container with ID starting with 1a552d8ed5626d1ac8c44a7752ccc3824002233d82a36a71a0b1e79ded6283dc not found: ID does not exist" Oct 30 06:05:45.332398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.332324 5980 scope.go:122] "RemoveContainer" containerID="2cff76b764d09b394473cac8665d275dabecbe579539474fc133525da4c54fcb" Oct 30 06:05:45.350079 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.350049 5980 scope.go:122] "RemoveContainer" containerID="3d9531426d233f95eebf51ce413a51bac8832676b99d67568f4ce4d66ebaaed9" Oct 30 06:05:45.372944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.372902 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.372944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.372942 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.373185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.372956 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7sf4\" (UniqueName: \"kubernetes.io/projected/afd53c91-9a4c-4243-a657-ce77ddd88282-kube-api-access-x7sf4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.373185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.372970 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd53c91-9a4c-4243-a657-ce77ddd88282-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:45.412791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.412742 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7"] Oct 30 06:05:45.416963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.416926 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-k64k7"] Oct 30 06:05:45.452330 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.452290 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh"] Oct 30 06:05:45.456100 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.456062 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-dj2hh"] Oct 30 06:05:45.493129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.493076 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x"] Oct 30 06:05:45.498458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:45.498404 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-4zf6x"] Oct 30 06:05:46.017434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:46.017396 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" path="/var/lib/kubelet/pods/61eb1e21-74fd-4bdf-9782-fa4a081782e3/volumes" Oct 30 06:05:46.017981 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:46.017965 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" path="/var/lib/kubelet/pods/afd53c91-9a4c-4243-a657-ce77ddd88282/volumes" Oct 30 06:05:46.018545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:46.018527 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b793ae0f-d086-4c38-af08-d68376033083" path="/var/lib/kubelet/pods/b793ae0f-d086-4c38-af08-d68376033083/volumes" Oct 30 06:05:50.573895 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:50.573847 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk"] Oct 30 06:05:50.574526 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:50.574082 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" containerID="cri-o://a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c" gracePeriod=30 Oct 30 06:05:55.877796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.877761 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 06:05:55.975075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975042 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975178 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975285 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975421 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975383 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975476 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:55.975567 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975491 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975604 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975764 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975729 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975782 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.975857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.975820 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h\") pod \"d40be304-8a11-48bd-aa34-94577f784fb1\" (UID: \"d40be304-8a11-48bd-aa34-94577f784fb1\") " Oct 30 06:05:55.977522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.977318 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:55.977522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.977348 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:55.977522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.977429 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:55.977522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.977490 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:55.978036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.978007 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:55.981404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.980500 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:55.982094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.981569 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:55.982436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.982410 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "kube-api-access-jrm9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:55.982836 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:55.982811 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token" pod "d40be304-8a11-48bd-aa34-94577f784fb1" (UID: "d40be304-8a11-48bd-aa34-94577f784fb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:56.079536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079499 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079534 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079545 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079556 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrm9h\" (UniqueName: \"kubernetes.io/projected/d40be304-8a11-48bd-aa34-94577f784fb1-kube-api-access-jrm9h\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079569 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40be304-8a11-48bd-aa34-94577f784fb1-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079581 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079592 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40be304-8a11-48bd-aa34-94577f784fb1-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.079794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.079602 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40be304-8a11-48bd-aa34-94577f784fb1-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:05:56.421718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.421678 5980 generic.go:356] "Generic (PLEG): container finished" podID="d40be304-8a11-48bd-aa34-94577f784fb1" containerID="a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c" exitCode=0 Oct 30 06:05:56.421966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.421734 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" Oct 30 06:05:56.421966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.421792 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" event={"ID":"d40be304-8a11-48bd-aa34-94577f784fb1","Type":"ContainerDied","Data":"a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c"} Oct 30 06:05:56.421966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.421864 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk" event={"ID":"d40be304-8a11-48bd-aa34-94577f784fb1","Type":"ContainerDied","Data":"b53c4e0ca0890aff8753d14f4dd544351d982314de81a30cb1af536f370834ec"} Oct 30 06:05:56.421966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.421900 5980 scope.go:122] "RemoveContainer" containerID="a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c" Oct 30 06:05:56.434677 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.434654 5980 scope.go:122] "RemoveContainer" containerID="a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c" Oct 30 06:05:56.435015 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:05:56.434995 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c\": container with ID starting with a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c not found: ID does not exist" containerID="a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c" Oct 30 06:05:56.435099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.435068 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c"} err="failed to get container status \"a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c\": rpc error: code = NotFound desc = could not find container \"a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c\": container with ID starting with a43f2e4be2d2828d25b3e56b6dd75bbed2e94296e7a7a9ba5ee5152bf6685e3c not found: ID does not exist" Oct 30 06:05:56.503217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.497397 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk"] Oct 30 06:05:56.526134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:56.524340 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-dvbqk"] Oct 30 06:05:58.018075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:05:58.018028 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" path="/var/lib/kubelet/pods/d40be304-8a11-48bd-aa34-94577f784fb1/volumes" Oct 30 06:06:00.218145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:00.212780 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.471194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:00.465349 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.641136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:00.640430 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.643945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:00.643221 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:13.104330 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:06:13.104286 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:06:15.545343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:15.545297 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:06:15.546031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:15.545376 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:18.760860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:18.760759 5980 generic.go:356] "Generic (PLEG): container finished" podID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerID="20460ffbf18a89958ceb88fb219f9982c443c5f492e794b51515649c1eebf3c4" exitCode=0 Oct 30 06:06:18.760860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:18.760840 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" event={"ID":"c53c62e9-60a6-44d4-94d1-9391b1b3a14a","Type":"ContainerDied","Data":"20460ffbf18a89958ceb88fb219f9982c443c5f492e794b51515649c1eebf3c4"} Oct 30 06:06:18.761434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:18.761409 5980 scope.go:122] "RemoveContainer" containerID="20460ffbf18a89958ceb88fb219f9982c443c5f492e794b51515649c1eebf3c4" Oct 30 06:06:18.763813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:18.763768 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:18.763980 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:18.763875 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:18.763980 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:18.763929 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:19.132208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.132165 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw"] Oct 30 06:06:19.133466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133435 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" Oct 30 06:06:19.133544 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133471 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" Oct 30 06:06:19.133582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133567 5980 cpu_manager.go:414] "containerMap: removing container" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" Oct 30 06:06:19.133582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133576 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" Oct 30 06:06:19.133650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133626 5980 cpu_manager.go:414] "containerMap: removing container" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" Oct 30 06:06:19.133650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133635 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" Oct 30 06:06:19.133650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133648 5980 cpu_manager.go:414] "containerMap: removing container" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" Oct 30 06:06:19.133749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133655 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" Oct 30 06:06:19.133862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133846 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="61eb1e21-74fd-4bdf-9782-fa4a081782e3" containerName="istio-proxy" Oct 30 06:06:19.133915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133871 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40be304-8a11-48bd-aa34-94577f784fb1" containerName="istio-proxy" Oct 30 06:06:19.133915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133893 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd53c91-9a4c-4243-a657-ce77ddd88282" containerName="istio-proxy" Oct 30 06:06:19.133915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.133904 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b793ae0f-d086-4c38-af08-d68376033083" containerName="istio-proxy" Oct 30 06:06:19.137624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.137580 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.181178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.180404 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw"] Oct 30 06:06:19.186610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.186562 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.186829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.186711 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.186829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.186799 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.186945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.186831 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.186945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.186887 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxflz\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.186945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.186930 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.187212 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.187082 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.187212 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.187202 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.187348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.187270 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.189082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.189046 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-03d349-openshift-default-dockercfg-2zmxp\"" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337267 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337390 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337499 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337577 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337607 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337658 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxflz\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337697 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337845 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.337940 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.339717 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.340026 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.342221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.340796 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.400203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.386419 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.400203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.386822 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.400203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.390951 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxflz\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.400203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.395458 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.428528 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.407775 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.452136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.448499 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.462152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.460003 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:19.783810 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:19.783764 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" event={"ID":"c53c62e9-60a6-44d4-94d1-9391b1b3a14a","Type":"ContainerStarted","Data":"646642e53ffc0108c477fc5b1d8e1dd0923d8f66f44b46e09d85b968e051bf29"} Oct 30 06:06:20.044079 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.042439 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw"] Oct 30 06:06:20.046052 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:06:20.046008 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7294cfd8_d116_49bf_9361_a2778bea6118.slice/crio-a838e330e4350b9fa8b06a630b9cb74958d7ab21f6a4608441e800039b52cc65 WatchSource:0}: Error finding container a838e330e4350b9fa8b06a630b9cb74958d7ab21f6a4608441e800039b52cc65: Status 404 returned error can't find the container with id a838e330e4350b9fa8b06a630b9cb74958d7ab21f6a4608441e800039b52cc65 Oct 30 06:06:20.050083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.050024 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:20.050286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.050161 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:20.050286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.050243 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:20.612780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.612742 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:06:20.613119 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.613074 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:20.613219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.613193 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:20.805946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.805898 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" event={"ID":"7294cfd8-d116-49bf-9361-a2778bea6118","Type":"ContainerStarted","Data":"f30dc64e597392af6741ec10a007d9797e9130d295b5493f118a88de3d08ebb5"} Oct 30 06:06:20.806620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.805981 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" event={"ID":"7294cfd8-d116-49bf-9361-a2778bea6118","Type":"ContainerStarted","Data":"a838e330e4350b9fa8b06a630b9cb74958d7ab21f6a4608441e800039b52cc65"} Oct 30 06:06:20.856948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:20.856882 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podStartSLOduration=1.856864048 podStartE2EDuration="1.856864048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:20.85504308 +0000 UTC m=+2034.963082113" watchObservedRunningTime="2026-10-30 06:06:20.856864048 +0000 UTC m=+2034.964903080" Oct 30 06:06:21.461058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:21.461010 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:21.498531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:21.498496 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:21.613325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:21.613284 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:21.613531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:21.613355 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:21.815155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:21.815039 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:21.816332 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:21.816308 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:06:22.294238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.294184 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw"] Oct 30 06:06:22.309611 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.309576 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw"] Oct 30 06:06:22.309806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.309682 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.334849 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.334802 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d64c9-openshift-default-dockercfg-zxnwz\"" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454479 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454532 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454562 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454584 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454682 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454722 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvrv\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454751 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454808 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.457610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.454881 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556396 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556300 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556396 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556351 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556635 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556501 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556681 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556757 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvrv\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556813 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556906 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556814 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.556962 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.556905 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.557064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.557039 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.557167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.557059 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.557263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.557243 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.557537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.557505 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.557681 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.557553 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.557752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.557689 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.558977 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.558959 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.559829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.559810 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.610038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.610000 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.612149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.612096 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvrv\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:22.613487 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.613458 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:22.613693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.613670 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:22.631596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:22.631559 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:23.009677 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.008665 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5"] Oct 30 06:06:23.014406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.014380 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.036620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.036583 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5"] Oct 30 06:06:23.042207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.042180 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-62316c-openshift-default-dockercfg-t276w\"" Oct 30 06:06:23.050692 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.050611 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw"] Oct 30 06:06:23.052211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.052152 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:23.052351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.052259 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:23.052351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.052308 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207452 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207547 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207657 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207725 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkrd\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207773 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207810 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207826 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207873 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.208137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.207992 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309295 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkrd\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309393 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309441 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309473 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309544 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309650 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309754 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309856 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.309963 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.310452 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.311377 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.316780 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.317515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.317041 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.322574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.322538 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.339853 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.339786 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.343473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.343443 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.351155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.348263 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkrd\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.351155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.348314 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.616537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.616379 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:23.616537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.616444 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:23.635231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.635200 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:23.858264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.858209 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" event={"ID":"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb","Type":"ContainerStarted","Data":"08a50b0d4a629cf3597a740b4180343a2d44244cee53c7568f1f475f869da7c4"} Oct 30 06:06:23.858484 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.858336 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" event={"ID":"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb","Type":"ContainerStarted","Data":"2ab29e0714625c706f2168f367e8800058a1cd036342dcc3b77550bc3a59cf96"} Oct 30 06:06:23.946463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:23.945434 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podStartSLOduration=1.945409857 podStartE2EDuration="1.945409857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:23.927016102 +0000 UTC m=+2038.035055134" watchObservedRunningTime="2026-10-30 06:06:23.945409857 +0000 UTC m=+2038.053448892" Oct 30 06:06:24.052418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.051826 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5"] Oct 30 06:06:24.055347 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:06:24.055308 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf3a979_6693_479c_9f19_0547f62be41b.slice/crio-b3211d294539e1b30562e1db5b673d277c1b2b483e7d9253b0dd19041cb5f848 WatchSource:0}: Error finding container b3211d294539e1b30562e1db5b673d277c1b2b483e7d9253b0dd19041cb5f848: Status 404 returned error can't find the container with id b3211d294539e1b30562e1db5b673d277c1b2b483e7d9253b0dd19041cb5f848 Oct 30 06:06:24.059098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.059055 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:24.059380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.059360 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:24.059481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.059468 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:24.640158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.618365 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:24.640158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.618446 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:24.656327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.656216 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:24.676708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.676664 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:24.677017 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.676973 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:24.930894 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.930796 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" event={"ID":"cdf3a979-6693-479c-9f19-0547f62be41b","Type":"ContainerStarted","Data":"67185d812822c29b7fba9fb4f0e7db6315d882240c1afef0291db773aea47e03"} Oct 30 06:06:24.931180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:24.931153 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" event={"ID":"cdf3a979-6693-479c-9f19-0547f62be41b","Type":"ContainerStarted","Data":"b3211d294539e1b30562e1db5b673d277c1b2b483e7d9253b0dd19041cb5f848"} Oct 30 06:06:25.047188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.042372 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podStartSLOduration=3.042349901 podStartE2EDuration="3.042349901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:25.039523313 +0000 UTC m=+2039.147562346" watchObservedRunningTime="2026-10-30 06:06:25.042349901 +0000 UTC m=+2039.150388935" Oct 30 06:06:25.630133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.629248 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:25.630133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.629316 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:25.659205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.648474 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:25.659205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.648541 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:25.677318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.649187 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:25.764414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.751358 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:25.764414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:25.751428 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:26.557131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.556285 5980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:26.620136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.614474 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5" podUID="206d662f-b8a8-48b5-b8f9-c74d9785870f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Oct 30 06:06:26.620136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.616831 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:26.620136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.617548 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 06:06:26.620136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.617597 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 06:06:26.640132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.638802 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:26.640132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.638863 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:26.640132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.639163 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:26.640132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.639196 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:26.652084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:26.651903 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:27.635084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.634232 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:27.635084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.634339 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:27.637138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.636456 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:27.637138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.636528 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:27.641280 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.641243 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:06:27.642264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.642237 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:06:27.643448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:27.643424 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:06:28.634176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:28.634093 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:28.634439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:28.634195 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:28.643155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:28.642598 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:28.643155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:28.642671 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:29.820502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:29.810805 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:29.820502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:29.810869 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:29.820502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:29.819739 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:29.820502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:29.819794 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:30.635025 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:30.634270 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:30.635025 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:30.634420 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:30.636087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:30.636058 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:30.636422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:30.636394 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:31.633680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:31.633635 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:06:31.634257 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:31.633699 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:06:31.636697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:31.636664 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:31.636873 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:31.636716 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:32.636014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:32.635975 5980 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 06:06:32.636521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:32.636053 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 06:06:32.640922 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:32.640893 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:32.641388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:32.641366 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:32.642219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:32.642195 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:06:33.643014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:33.642982 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:33.643508 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:33.643474 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:33.644472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:33.644451 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:06:35.474066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.474026 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc"] Oct 30 06:06:35.479221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.479193 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:35.499644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.499606 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc"] Oct 30 06:06:35.508399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.508356 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0d0152\"" Oct 30 06:06:35.566942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.566899 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr"] Oct 30 06:06:35.570882 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.570855 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:35.708710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.604914 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0d0152-dockercfg-cmjgp\"" Oct 30 06:06:35.717546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.716715 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:35.717546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.716904 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljwf\" (UniqueName: \"kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf\") pod \"payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:35.717546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.717059 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwk7\" (UniqueName: \"kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7\") pod \"payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:35.717546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.717270 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:35.880372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.880322 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr"] Oct 30 06:06:35.945676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.945637 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:36.002234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.002193 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:36.002730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.002653 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljwf\" (UniqueName: \"kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf\") pod \"payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:36.003161 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:35.952977 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:36.003738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.003711 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:36.004252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.004216 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwk7\" (UniqueName: \"kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7\") pod \"payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:36.037141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.036874 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljwf\" (UniqueName: \"kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf\") pod \"payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:36.042153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.038413 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwk7\" (UniqueName: \"kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7\") pod \"payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:36.097751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.097710 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:36.193309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.192704 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:36.438412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.438377 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc"] Oct 30 06:06:36.438412 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:06:36.438393 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c625ed_ad61_4c28_9e97_db1fde32c265.slice/crio-8ebed4bfc9450c20d61ec2154b8abaf20599b1e61a46226e7818c0303123150f WatchSource:0}: Error finding container 8ebed4bfc9450c20d61ec2154b8abaf20599b1e61a46226e7818c0303123150f: Status 404 returned error can't find the container with id 8ebed4bfc9450c20d61ec2154b8abaf20599b1e61a46226e7818c0303123150f Oct 30 06:06:36.549430 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:06:36.549398 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff32e94e_a308_4306_b0f0_75c67f6832d7.slice/crio-c78611928d77df678e8a5786c8809027962eea86286f88452c4bdd4d0138c811 WatchSource:0}: Error finding container c78611928d77df678e8a5786c8809027962eea86286f88452c4bdd4d0138c811: Status 404 returned error can't find the container with id c78611928d77df678e8a5786c8809027962eea86286f88452c4bdd4d0138c811 Oct 30 06:06:36.549847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:36.549540 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr"] Oct 30 06:06:37.272461 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.272422 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" event={"ID":"87c625ed-ad61-4c28-9e97-db1fde32c265","Type":"ContainerStarted","Data":"11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c"} Oct 30 06:06:37.272676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.272479 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" event={"ID":"87c625ed-ad61-4c28-9e97-db1fde32c265","Type":"ContainerStarted","Data":"8ebed4bfc9450c20d61ec2154b8abaf20599b1e61a46226e7818c0303123150f"} Oct 30 06:06:37.272676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.272659 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:06:37.274993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.274954 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" event={"ID":"ff32e94e-a308-4306-b0f0-75c67f6832d7","Type":"ContainerStarted","Data":"30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771"} Oct 30 06:06:37.275251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.275021 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" event={"ID":"ff32e94e-a308-4306-b0f0-75c67f6832d7","Type":"ContainerStarted","Data":"c78611928d77df678e8a5786c8809027962eea86286f88452c4bdd4d0138c811"} Oct 30 06:06:37.275251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.275164 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:37.317457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.317390 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" podStartSLOduration=2.317371274 podStartE2EDuration="2.317371274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:37.315237985 +0000 UTC m=+2051.423277019" watchObservedRunningTime="2026-10-30 06:06:37.317371274 +0000 UTC m=+2051.425410342" Oct 30 06:06:37.361820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:37.361707 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" podStartSLOduration=2.3616879219999998 podStartE2EDuration="2.361687922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:37.358124594 +0000 UTC m=+2051.466163625" watchObservedRunningTime="2026-10-30 06:06:37.361687922 +0000 UTC m=+2051.469726984" Oct 30 06:06:43.105672 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:06:43.105622 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:06:48.289366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:48.289326 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:06:48.289897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:06:48.289419 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:07:13.107750 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:07:13.107674 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:07:38.086732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.086694 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7"] Oct 30 06:07:38.092612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.092579 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.111865 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.111827 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c54211-openshift-default-dockercfg-t6b72\"" Oct 30 06:07:38.129678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.129639 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7"] Oct 30 06:07:38.264972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.264910 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.264972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.264978 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265350 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265298 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265366 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265413 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265469 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265531 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265563 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qvq\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.265713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.265703 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.372577 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.372751 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qvq\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.372965 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.373134 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.373167 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.373319 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.373361 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.373387 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.377081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.373421 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.386564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.386514 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.387656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.387622 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.387934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.387910 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.388048 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.386523 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.388590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.388567 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.406045 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.405276 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.406045 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.405596 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.433070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.432959 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qvq\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.439187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.438461 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-4rdl7\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:38.712032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:38.711934 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:39.068174 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:07:39.068082 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5dc528_308f_4d4b_9484_1edbaa20c8b6.slice/crio-c29783c12bc904ecdf0f1d19e141a1b3444123084d29777034617950a35663be WatchSource:0}: Error finding container c29783c12bc904ecdf0f1d19e141a1b3444123084d29777034617950a35663be: Status 404 returned error can't find the container with id c29783c12bc904ecdf0f1d19e141a1b3444123084d29777034617950a35663be Oct 30 06:07:39.068425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.068396 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7"] Oct 30 06:07:39.071776 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.071738 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:39.071933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.071816 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:39.071933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.071850 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:39.340018 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.339977 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" event={"ID":"ef5dc528-308f-4d4b-9484-1edbaa20c8b6","Type":"ContainerStarted","Data":"edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0"} Oct 30 06:07:39.340614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.340047 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" event={"ID":"ef5dc528-308f-4d4b-9484-1edbaa20c8b6","Type":"ContainerStarted","Data":"c29783c12bc904ecdf0f1d19e141a1b3444123084d29777034617950a35663be"} Oct 30 06:07:39.395840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.395766 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podStartSLOduration=1.395742189 podStartE2EDuration="1.395742189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:39.389523315 +0000 UTC m=+2113.497562349" watchObservedRunningTime="2026-10-30 06:07:39.395742189 +0000 UTC m=+2113.503781222" Oct 30 06:07:39.712122 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:39.711997 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:40.715398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:40.715358 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 06:07:40.715966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:40.715446 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 06:07:41.712461 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:41.712424 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 06:07:41.712657 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:41.712493 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 06:07:42.713020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:42.712982 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 06:07:42.713483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:42.713040 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 06:07:43.126325 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:07:43.126283 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:07:43.724089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:43.724054 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 06:07:43.724745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:43.724711 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 06:07:44.423058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:44.423018 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg"] Oct 30 06:07:44.725194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:44.719905 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 06:07:44.725194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:44.719967 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 06:07:45.713020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:45.712983 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 06:07:45.713266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:45.713045 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 06:07:46.720851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:46.720817 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:46.722759 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:46.722725 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:46.724390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:46.724365 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:07:59.841743 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:07:59.739448 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerName="main" containerID="cri-o://4116ff2498c08d51c6cd10390b9473f2706cf9dee9f9a4d20f8bb07cff93e331" gracePeriod=45 Oct 30 06:08:00.809709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:00.809656 5980 generic.go:356] "Generic (PLEG): container finished" podID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerID="4116ff2498c08d51c6cd10390b9473f2706cf9dee9f9a4d20f8bb07cff93e331" exitCode=0 Oct 30 06:08:00.809978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:00.809712 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": dial tcp 10.134.0.123:8000: connect: connection refused" Oct 30 06:08:00.809978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:00.809650 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" event={"ID":"76fa82d8-d669-4aeb-acab-6f9714712db5","Type":"ContainerDied","Data":"4116ff2498c08d51c6cd10390b9473f2706cf9dee9f9a4d20f8bb07cff93e331"} Oct 30 06:08:01.742444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:01.742402 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.123:8000/ready\": dial tcp 10.134.0.123:8000: connect: connection refused" Oct 30 06:08:01.742914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:01.742568 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:08:02.189496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.189454 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.292344 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq\" (UniqueName: \"kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq\") pod \"76fa82d8-d669-4aeb-acab-6f9714712db5\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.292580 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache\") pod \"76fa82d8-d669-4aeb-acab-6f9714712db5\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.292705 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home\") pod \"76fa82d8-d669-4aeb-acab-6f9714712db5\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.292849 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir\") pod \"76fa82d8-d669-4aeb-acab-6f9714712db5\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.292898 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm\") pod \"76fa82d8-d669-4aeb-acab-6f9714712db5\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.292963 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs\" (UniqueName: \"kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs\") pod \"76fa82d8-d669-4aeb-acab-6f9714712db5\" (UID: \"76fa82d8-d669-4aeb-acab-6f9714712db5\") " Oct 30 06:08:02.294449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.294435 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir" pod "76fa82d8-d669-4aeb-acab-6f9714712db5" (UID: "76fa82d8-d669-4aeb-acab-6f9714712db5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:02.297697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.294787 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home" pod "76fa82d8-d669-4aeb-acab-6f9714712db5" (UID: "76fa82d8-d669-4aeb-acab-6f9714712db5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:02.297697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.295008 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache" pod "76fa82d8-d669-4aeb-acab-6f9714712db5" (UID: "76fa82d8-d669-4aeb-acab-6f9714712db5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:02.331219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.331148 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs" pod "76fa82d8-d669-4aeb-acab-6f9714712db5" (UID: "76fa82d8-d669-4aeb-acab-6f9714712db5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:02.331219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.331169 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm" pod "76fa82d8-d669-4aeb-acab-6f9714712db5" (UID: "76fa82d8-d669-4aeb-acab-6f9714712db5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:02.331219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.331209 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq" pod "76fa82d8-d669-4aeb-acab-6f9714712db5" (UID: "76fa82d8-d669-4aeb-acab-6f9714712db5"). InnerVolumeSpecName "kube-api-access-jknlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:02.404150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.402662 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.404150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.402703 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.404150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.402716 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fa82d8-d669-4aeb-acab-6f9714712db5-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.404150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.402740 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jknlq\" (UniqueName: \"kubernetes.io/projected/76fa82d8-d669-4aeb-acab-6f9714712db5-kube-api-access-jknlq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.404150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.402756 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.404150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.402769 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fa82d8-d669-4aeb-acab-6f9714712db5-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.830271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.830233 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" event={"ID":"76fa82d8-d669-4aeb-acab-6f9714712db5","Type":"ContainerDied","Data":"2b216c88f3cfe5acad4e0543e3bc70e3cc5585956a9f0c90289501155a42e050"} Oct 30 06:08:02.830844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.830307 5980 scope.go:122] "RemoveContainer" containerID="4116ff2498c08d51c6cd10390b9473f2706cf9dee9f9a4d20f8bb07cff93e331" Oct 30 06:08:02.830844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.830245 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg" Oct 30 06:08:02.892235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.892196 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg"] Oct 30 06:08:02.905530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:02.905478 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-387b16/test-model-387b16-kserve-68b5765597-c2qtg"] Oct 30 06:08:03.841479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:03.841442 5980 generic.go:356] "Generic (PLEG): container finished" podID="7294cfd8-d116-49bf-9361-a2778bea6118" containerID="f30dc64e597392af6741ec10a007d9797e9130d295b5493f118a88de3d08ebb5" exitCode=0 Oct 30 06:08:03.842429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:03.841505 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" event={"ID":"7294cfd8-d116-49bf-9361-a2778bea6118","Type":"ContainerDied","Data":"f30dc64e597392af6741ec10a007d9797e9130d295b5493f118a88de3d08ebb5"} Oct 30 06:08:03.842429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:03.842061 5980 scope.go:122] "RemoveContainer" containerID="f30dc64e597392af6741ec10a007d9797e9130d295b5493f118a88de3d08ebb5" Oct 30 06:08:03.844588 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:03.844536 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:03.844723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:03.844638 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:03.844723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:03.844674 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:04.022807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:04.022767 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" path="/var/lib/kubelet/pods/76fa82d8-d669-4aeb-acab-6f9714712db5/volumes" Oct 30 06:08:04.866758 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:04.866716 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" event={"ID":"7294cfd8-d116-49bf-9361-a2778bea6118","Type":"ContainerStarted","Data":"3c782c1ce45a2bd66c0c92cd7fe65ea805f1c59448895a6f5ffe5d41ed15bfff"} Oct 30 06:08:05.461040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:05.460997 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:08:05.480134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:05.464263 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:08:05.480134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:05.464348 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:08:06.460858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:06.460817 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:08:06.461284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:06.460883 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:08:06.815772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:06.815087 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:08:07.462151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:07.461152 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:08:07.462151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:07.461220 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:08:08.463131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:08.462228 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:08:08.463131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:08.462286 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:08:09.461059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:09.461015 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:08:09.462001 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:09.461576 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:08:10.461151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:10.461079 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:08:10.461608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:10.461214 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:08:11.468221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:11.468180 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:08:11.471020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:11.470988 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:08:13.128727 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:08:13.128680 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:08:24.500042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:24.499989 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7"] Oct 30 06:08:25.913325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:25.913282 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd"] Oct 30 06:08:25.913875 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:25.913552 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" podUID="debeaae1-7065-4605-bc17-0cf35365f795" containerName="maas-api" containerID="cri-o://fb05e3c0d84b092c49461350706de79ef7be63be889031bea54442fc9e1c5e5e" gracePeriod=30 Oct 30 06:08:26.475933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.475708 5980 generic.go:356] "Generic (PLEG): container finished" podID="debeaae1-7065-4605-bc17-0cf35365f795" containerID="fb05e3c0d84b092c49461350706de79ef7be63be889031bea54442fc9e1c5e5e" exitCode=0 Oct 30 06:08:26.475933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.475858 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" event={"ID":"debeaae1-7065-4605-bc17-0cf35365f795","Type":"ContainerDied","Data":"fb05e3c0d84b092c49461350706de79ef7be63be889031bea54442fc9e1c5e5e"} Oct 30 06:08:26.572688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.570617 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:08:26.713443 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.712812 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n\" (UniqueName: \"kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n\") pod \"debeaae1-7065-4605-bc17-0cf35365f795\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " Oct 30 06:08:26.713443 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.713434 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls\") pod \"debeaae1-7065-4605-bc17-0cf35365f795\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " Oct 30 06:08:26.713842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.713811 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs\") pod \"debeaae1-7065-4605-bc17-0cf35365f795\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " Oct 30 06:08:26.714384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.714335 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca\") pod \"debeaae1-7065-4605-bc17-0cf35365f795\" (UID: \"debeaae1-7065-4605-bc17-0cf35365f795\") " Oct 30 06:08:26.714835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.714735 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca" pod "debeaae1-7065-4605-bc17-0cf35365f795" (UID: "debeaae1-7065-4605-bc17-0cf35365f795"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:26.716046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.715875 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debeaae1-7065-4605-bc17-0cf35365f795-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:26.742194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.735006 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls" pod "debeaae1-7065-4605-bc17-0cf35365f795" (UID: "debeaae1-7065-4605-bc17-0cf35365f795"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:26.770148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.765345 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n" pod "debeaae1-7065-4605-bc17-0cf35365f795" (UID: "debeaae1-7065-4605-bc17-0cf35365f795"). InnerVolumeSpecName "kube-api-access-t6c4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:26.787133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.783359 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs" pod "debeaae1-7065-4605-bc17-0cf35365f795" (UID: "debeaae1-7065-4605-bc17-0cf35365f795"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:26.817774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.817737 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:26.817774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.817773 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6c4n\" (UniqueName: \"kubernetes.io/projected/debeaae1-7065-4605-bc17-0cf35365f795-kube-api-access-t6c4n\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:26.817774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:26.817784 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debeaae1-7065-4605-bc17-0cf35365f795-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:27.184506 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:27.184465 5980 scope.go:122] "RemoveContainer" containerID="fb05e3c0d84b092c49461350706de79ef7be63be889031bea54442fc9e1c5e5e" Oct 30 06:08:27.490595 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:27.490496 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" event={"ID":"debeaae1-7065-4605-bc17-0cf35365f795","Type":"ContainerDied","Data":"6c66929250731df5ad4e8171b2d324d1c467f3b43702d83c31cb3dca65579278"} Oct 30 06:08:27.490758 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:27.490592 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd" Oct 30 06:08:27.580850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:27.580813 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd"] Oct 30 06:08:27.587166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:27.587073 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-387b16-7fb98899fb-4pchd"] Oct 30 06:08:28.026425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:28.026390 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debeaae1-7065-4605-bc17-0cf35365f795" path="/var/lib/kubelet/pods/debeaae1-7065-4605-bc17-0cf35365f795/volumes" Oct 30 06:08:29.500419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:29.500367 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerName="payload-pre-processing" containerID="cri-o://108ecc79d3669b810d39065ae4e8a422974549f6689d1fc1ea53b7dfb0f5143f" gracePeriod=25 Oct 30 06:08:34.194970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:34.194925 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc"] Oct 30 06:08:34.195974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:34.195935 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" containerID="cri-o://d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9" gracePeriod=30 Oct 30 06:08:34.871469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:34.871431 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:37.455306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:37.455220 5980 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:08:37.455707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:37.455298 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:08:39.332416 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:08:39.332375 5980 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc203a63b_accb_4462_88a0_1b3dc81fa87b.slice/crio-d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc203a63b_accb_4462_88a0_1b3dc81fa87b.slice/crio-conmon-d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:08:39.505069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.505037 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:08:39.586223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586186 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586241 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586358 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586459 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586509 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586669 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586582 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586738 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586831 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.586957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.586890 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2\") pod \"c203a63b-accb-4462-88a0-1b3dc81fa87b\" (UID: \"c203a63b-accb-4462-88a0-1b3dc81fa87b\") " Oct 30 06:08:39.587547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.587318 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.587547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.587390 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.588215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.587854 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.588215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.587881 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.588215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.587921 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:39.588215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.587925 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.588554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.588520 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.613196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.613048 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:39.613347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.613290 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "kube-api-access-k86b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.614819 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.614796 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.614910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.614830 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy" pod "c203a63b-accb-4462-88a0-1b3dc81fa87b" (UID: "c203a63b-accb-4462-88a0-1b3dc81fa87b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.674957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.674917 5980 generic.go:356] "Generic (PLEG): container finished" podID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerID="d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9" exitCode=0 Oct 30 06:08:39.675170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.674959 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" event={"ID":"c203a63b-accb-4462-88a0-1b3dc81fa87b","Type":"ContainerDied","Data":"d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9"} Oct 30 06:08:39.675170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.674994 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" Oct 30 06:08:39.675170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.675011 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc" event={"ID":"c203a63b-accb-4462-88a0-1b3dc81fa87b","Type":"ContainerDied","Data":"d0adf62fc2c416fca90400be04cc57ab126c0908e50d23b9e0e65ca609d62ed7"} Oct 30 06:08:39.675170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.675038 5980 scope.go:122] "RemoveContainer" containerID="d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.689988 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c203a63b-accb-4462-88a0-1b3dc81fa87b-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.690014 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.690027 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.690041 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.690054 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c203a63b-accb-4462-88a0-1b3dc81fa87b-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.690066 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.690366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.690081 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86b2\" (UniqueName: \"kubernetes.io/projected/c203a63b-accb-4462-88a0-1b3dc81fa87b-kube-api-access-k86b2\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.718611 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.718587 5980 scope.go:122] "RemoveContainer" containerID="d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9" Oct 30 06:08:39.718982 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:08:39.718956 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9\": container with ID starting with d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9 not found: ID does not exist" containerID="d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9" Oct 30 06:08:39.719058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.718989 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9"} err="failed to get container status \"d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9\": rpc error: code = NotFound desc = could not find container \"d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9\": container with ID starting with d8270e61aa1cdf011d859bde7e868c7099b18b8434ea6f0e4aa1cbf66ca9ada9 not found: ID does not exist" Oct 30 06:08:39.739772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.739736 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc"] Oct 30 06:08:39.746238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:39.746195 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-wscbc"] Oct 30 06:08:40.016712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:40.016634 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" path="/var/lib/kubelet/pods/c203a63b-accb-4462-88a0-1b3dc81fa87b/volumes" Oct 30 06:08:43.095501 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:08:43.095447 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:08:44.870432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:44.870393 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:45.157324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:45.157211 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z"] Oct 30 06:08:45.191262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:45.191218 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk"] Oct 30 06:08:50.038157 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:50.036877 5980 generic.go:356] "Generic (PLEG): container finished" podID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerID="108ecc79d3669b810d39065ae4e8a422974549f6689d1fc1ea53b7dfb0f5143f" exitCode=0 Oct 30 06:08:50.071158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:50.070672 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" event={"ID":"968c465c-a936-4ca0-acd4-10c7aa3dc2fd","Type":"ContainerDied","Data":"108ecc79d3669b810d39065ae4e8a422974549f6689d1fc1ea53b7dfb0f5143f"} Oct 30 06:08:50.158444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:50.158215 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" podUID="ef86243d-038b-45e6-95ee-aa4a2f279bde" containerName="payload-processing" containerID="cri-o://3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85" gracePeriod=25 Oct 30 06:08:50.195758 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:50.195523 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" podUID="cbdf8e63-5588-49de-a008-b864cd633447" containerName="payload-pre-processing" containerID="cri-o://971afb3d24fd57438cc910caf3b741fadba81331ef1b8579db442d2e312c0bd8" gracePeriod=25 Oct 30 06:08:50.903327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:50.903294 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:08:51.010404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.010372 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:08:51.048390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.048350 5980 generic.go:356] "Generic (PLEG): container finished" podID="ef86243d-038b-45e6-95ee-aa4a2f279bde" containerID="3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85" exitCode=0 Oct 30 06:08:51.048921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.048399 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" Oct 30 06:08:51.048921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.048438 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" event={"ID":"ef86243d-038b-45e6-95ee-aa4a2f279bde","Type":"ContainerDied","Data":"3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85"} Oct 30 06:08:51.048921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.048492 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z" event={"ID":"ef86243d-038b-45e6-95ee-aa4a2f279bde","Type":"ContainerDied","Data":"3f02244c1cc6c814943756bf9ef853ad000befbfe087f5eefb703799d49c37dd"} Oct 30 06:08:51.048921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.048520 5980 scope.go:122] "RemoveContainer" containerID="3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85" Oct 30 06:08:51.050036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.050018 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" Oct 30 06:08:51.050233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.050060 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7" event={"ID":"968c465c-a936-4ca0-acd4-10c7aa3dc2fd","Type":"ContainerDied","Data":"6f72d624c127b6ac3e81bce69e42ba6eb3965e2a6262c86bdded1e91dffb90e5"} Oct 30 06:08:51.056944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.055531 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq\" (UniqueName: \"kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq\") pod \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " Oct 30 06:08:51.056944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.055934 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume\") pod \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\" (UID: \"968c465c-a936-4ca0-acd4-10c7aa3dc2fd\") " Oct 30 06:08:51.056944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.056347 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume" pod "968c465c-a936-4ca0-acd4-10c7aa3dc2fd" (UID: "968c465c-a936-4ca0-acd4-10c7aa3dc2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:51.056944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.056774 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.061945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.058827 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq" pod "968c465c-a936-4ca0-acd4-10c7aa3dc2fd" (UID: "968c465c-a936-4ca0-acd4-10c7aa3dc2fd"). InnerVolumeSpecName "kube-api-access-7rcdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:51.067566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.067543 5980 scope.go:122] "RemoveContainer" containerID="3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85" Oct 30 06:08:51.068100 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:08:51.068071 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85\": container with ID starting with 3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85 not found: ID does not exist" containerID="3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85" Oct 30 06:08:51.068100 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.068131 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85"} err="failed to get container status \"3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85\": rpc error: code = NotFound desc = could not find container \"3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85\": container with ID starting with 3dd2eda87d50309c009db9b5b0b3f2cbfe799c070d838db4adf31130ec635d85 not found: ID does not exist" Oct 30 06:08:51.068314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.068158 5980 scope.go:122] "RemoveContainer" containerID="108ecc79d3669b810d39065ae4e8a422974549f6689d1fc1ea53b7dfb0f5143f" Oct 30 06:08:51.190250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.187214 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5\" (UniqueName: \"kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5\") pod \"ef86243d-038b-45e6-95ee-aa4a2f279bde\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " Oct 30 06:08:51.190250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.187323 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume\") pod \"ef86243d-038b-45e6-95ee-aa4a2f279bde\" (UID: \"ef86243d-038b-45e6-95ee-aa4a2f279bde\") " Oct 30 06:08:51.190250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.188123 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rcdq\" (UniqueName: \"kubernetes.io/projected/968c465c-a936-4ca0-acd4-10c7aa3dc2fd-kube-api-access-7rcdq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.190250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.188548 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume" pod "ef86243d-038b-45e6-95ee-aa4a2f279bde" (UID: "ef86243d-038b-45e6-95ee-aa4a2f279bde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:51.236739 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.236690 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5" pod "ef86243d-038b-45e6-95ee-aa4a2f279bde" (UID: "ef86243d-038b-45e6-95ee-aa4a2f279bde"). InnerVolumeSpecName "kube-api-access-jn2x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:51.289744 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.289704 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2x5\" (UniqueName: \"kubernetes.io/projected/ef86243d-038b-45e6-95ee-aa4a2f279bde-kube-api-access-jn2x5\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.289744 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.289736 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef86243d-038b-45e6-95ee-aa4a2f279bde-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.403323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.403277 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7"] Oct 30 06:08:51.440138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.437642 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-3cc570-d49b598dc-pxlp7"] Oct 30 06:08:51.473216 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.473186 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z"] Oct 30 06:08:51.491777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:51.491729 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7ad9ad-54ff58c567-k6k5z"] Oct 30 06:08:52.018062 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:52.018024 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" path="/var/lib/kubelet/pods/968c465c-a936-4ca0-acd4-10c7aa3dc2fd/volumes" Oct 30 06:08:52.018614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:52.018599 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef86243d-038b-45e6-95ee-aa4a2f279bde" path="/var/lib/kubelet/pods/ef86243d-038b-45e6-95ee-aa4a2f279bde/volumes" Oct 30 06:08:56.701489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:08:56.701445 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" podUID="cbdf8e63-5588-49de-a008-b864cd633447" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:09:05.264902 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.264864 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb"] Oct 30 06:09:05.265583 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.265163 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" podUID="a2d9f949-38b1-4570-a219-5cc5649f5e4f" containerName="maas-api" containerID="cri-o://ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35" gracePeriod=30 Oct 30 06:09:05.876312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.876280 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:09:05.980068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.980020 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca\") pod \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " Oct 30 06:09:05.980294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.980086 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs\") pod \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " Oct 30 06:09:05.980294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.980251 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\") pod \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " Oct 30 06:09:05.980580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.980507 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b\" (UniqueName: \"kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b\") pod \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\" (UID: \"a2d9f949-38b1-4570-a219-5cc5649f5e4f\") " Oct 30 06:09:05.980757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.980706 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca" pod "a2d9f949-38b1-4570-a219-5cc5649f5e4f" (UID: "a2d9f949-38b1-4570-a219-5cc5649f5e4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:05.981450 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.981427 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d9f949-38b1-4570-a219-5cc5649f5e4f-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.983888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.983857 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b" pod "a2d9f949-38b1-4570-a219-5cc5649f5e4f" (UID: "a2d9f949-38b1-4570-a219-5cc5649f5e4f"). InnerVolumeSpecName "kube-api-access-hj56b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:05.984526 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.984468 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls" pod "a2d9f949-38b1-4570-a219-5cc5649f5e4f" (UID: "a2d9f949-38b1-4570-a219-5cc5649f5e4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:05.986658 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:05.986627 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs" pod "a2d9f949-38b1-4570-a219-5cc5649f5e4f" (UID: "a2d9f949-38b1-4570-a219-5cc5649f5e4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:06.095305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.090980 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:06.095305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.091014 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2d9f949-38b1-4570-a219-5cc5649f5e4f-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:06.095305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.091026 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj56b\" (UniqueName: \"kubernetes.io/projected/a2d9f949-38b1-4570-a219-5cc5649f5e4f-kube-api-access-hj56b\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:06.236136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.235486 5980 generic.go:356] "Generic (PLEG): container finished" podID="a2d9f949-38b1-4570-a219-5cc5649f5e4f" containerID="ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35" exitCode=0 Oct 30 06:09:06.236136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.235780 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" event={"ID":"a2d9f949-38b1-4570-a219-5cc5649f5e4f","Type":"ContainerDied","Data":"ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35"} Oct 30 06:09:06.236136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.235834 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" event={"ID":"a2d9f949-38b1-4570-a219-5cc5649f5e4f","Type":"ContainerDied","Data":"4783488cac2ce5088c6c96915ad7b9920cdde4dc167a138b0ce1817237181cb0"} Oct 30 06:09:06.236136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.235862 5980 scope.go:122] "RemoveContainer" containerID="ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35" Oct 30 06:09:06.236762 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.236170 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb" Oct 30 06:09:06.357399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.357349 5980 scope.go:122] "RemoveContainer" containerID="ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35" Oct 30 06:09:06.358248 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:09:06.358155 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35\": container with ID starting with ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35 not found: ID does not exist" containerID="ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35" Oct 30 06:09:06.358248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.358205 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35"} err="failed to get container status \"ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35\": rpc error: code = NotFound desc = could not find container \"ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35\": container with ID starting with ccd09135fd4be229da8fb8288a5bbed66a29afb64b12d511884be67644ac5e35 not found: ID does not exist" Oct 30 06:09:06.362437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.360983 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n"] Oct 30 06:09:06.362437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.361359 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" containerID="cri-o://cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9" gracePeriod=30 Oct 30 06:09:06.443229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.443173 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb"] Oct 30 06:09:06.451457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.451392 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aee070-5d5bcccc8f-nnsmb"] Oct 30 06:09:06.710465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.708802 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" podUID="cbdf8e63-5588-49de-a008-b864cd633447" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:09:06.958020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.957981 5980 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:06.958210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:06.958038 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:08.030214 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:08.030180 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d9f949-38b1-4570-a219-5cc5649f5e4f" path="/var/lib/kubelet/pods/a2d9f949-38b1-4570-a219-5cc5649f5e4f/volumes" Oct 30 06:09:10.277765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.277728 5980 generic.go:356] "Generic (PLEG): container finished" podID="cbdf8e63-5588-49de-a008-b864cd633447" containerID="971afb3d24fd57438cc910caf3b741fadba81331ef1b8579db442d2e312c0bd8" exitCode=0 Oct 30 06:09:10.278224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.277799 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" event={"ID":"cbdf8e63-5588-49de-a008-b864cd633447","Type":"ContainerDied","Data":"971afb3d24fd57438cc910caf3b741fadba81331ef1b8579db442d2e312c0bd8"} Oct 30 06:09:10.456815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.456738 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:09:10.574635 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.574588 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume\") pod \"cbdf8e63-5588-49de-a008-b864cd633447\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " Oct 30 06:09:10.574931 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.574913 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v\" (UniqueName: \"kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v\") pod \"cbdf8e63-5588-49de-a008-b864cd633447\" (UID: \"cbdf8e63-5588-49de-a008-b864cd633447\") " Oct 30 06:09:10.575036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.575005 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume" pod "cbdf8e63-5588-49de-a008-b864cd633447" (UID: "cbdf8e63-5588-49de-a008-b864cd633447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:10.575703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.575622 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbdf8e63-5588-49de-a008-b864cd633447-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:10.577616 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.577583 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v" pod "cbdf8e63-5588-49de-a008-b864cd633447" (UID: "cbdf8e63-5588-49de-a008-b864cd633447"). InnerVolumeSpecName "kube-api-access-95q6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:10.677599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:10.677557 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95q6v\" (UniqueName: \"kubernetes.io/projected/cbdf8e63-5588-49de-a008-b864cd633447-kube-api-access-95q6v\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.290164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.290077 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" Oct 30 06:09:11.290664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.290080 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk" event={"ID":"cbdf8e63-5588-49de-a008-b864cd633447","Type":"ContainerDied","Data":"97337af7f62ad1416ff2d634b40b56211e90e98942244345d928c3553d00a069"} Oct 30 06:09:11.290664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.290239 5980 scope.go:122] "RemoveContainer" containerID="971afb3d24fd57438cc910caf3b741fadba81331ef1b8579db442d2e312c0bd8" Oct 30 06:09:11.347987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.347952 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk"] Oct 30 06:09:11.361422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.361382 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7ad9ad-6c4f4d8659-l55wk"] Oct 30 06:09:11.778602 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.778567 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:09:11.904050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.903942 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904054 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904156 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904261 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904334 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904395 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.904483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904411 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.904483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904405 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904498 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904605 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.904935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.904812 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token\") pod \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\" (UID: \"9b028d8a-67dc-41fa-ba09-b4c59b3d7777\") " Oct 30 06:09:11.912152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.906901 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.912152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.906935 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.912152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.911464 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.912152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.911693 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.913923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.913829 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:11.914673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.914631 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.914877 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.914850 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "kube-api-access-2ttgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:11.915250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.915207 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:11.915421 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:11.915397 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo" pod "9b028d8a-67dc-41fa-ba09-b4c59b3d7777" (UID: "9b028d8a-67dc-41fa-ba09-b4c59b3d7777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:12.008698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008654 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.008698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008695 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.008698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008710 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.008972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008725 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.008972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008739 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ttgq\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-kube-api-access-2ttgq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.008972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008753 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.008972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.008766 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b028d8a-67dc-41fa-ba09-b4c59b3d7777-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.017225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.017185 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdf8e63-5588-49de-a008-b864cd633447" path="/var/lib/kubelet/pods/cbdf8e63-5588-49de-a008-b864cd633447/volumes" Oct 30 06:09:12.304823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.304699 5980 generic.go:356] "Generic (PLEG): container finished" podID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerID="cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9" exitCode=0 Oct 30 06:09:12.304823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.304764 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" event={"ID":"9b028d8a-67dc-41fa-ba09-b4c59b3d7777","Type":"ContainerDied","Data":"cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9"} Oct 30 06:09:12.305482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.304834 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" Oct 30 06:09:12.305482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.304846 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n" event={"ID":"9b028d8a-67dc-41fa-ba09-b4c59b3d7777","Type":"ContainerDied","Data":"a80dc21a199d05973d53e854722c8634dec42ca091a915d8a643613a589ba85a"} Oct 30 06:09:12.305482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.305278 5980 scope.go:122] "RemoveContainer" containerID="cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9" Oct 30 06:09:12.319334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.319302 5980 scope.go:122] "RemoveContainer" containerID="c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832" Oct 30 06:09:12.332066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.332039 5980 scope.go:122] "RemoveContainer" containerID="cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9" Oct 30 06:09:12.332454 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:09:12.332434 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9\": container with ID starting with cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9 not found: ID does not exist" containerID="cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9" Oct 30 06:09:12.332539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.332463 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9"} err="failed to get container status \"cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9\": rpc error: code = NotFound desc = could not find container \"cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9\": container with ID starting with cce4fd3239f20e97af6bbf9301c81f0d6bc0111b43712e78191a84306510e8e9 not found: ID does not exist" Oct 30 06:09:12.332539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.332483 5980 scope.go:122] "RemoveContainer" containerID="c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832" Oct 30 06:09:12.332801 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:09:12.332785 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832\": container with ID starting with c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832 not found: ID does not exist" containerID="c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832" Oct 30 06:09:12.332858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.332805 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832"} err="failed to get container status \"c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832\": rpc error: code = NotFound desc = could not find container \"c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832\": container with ID starting with c94b219d76731cd55e37f47de13d18477efbbd81a21de26b30f5f2635b83f832 not found: ID does not exist" Oct 30 06:09:12.363806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.363772 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n"] Oct 30 06:09:12.377065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:12.377023 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-5w67n"] Oct 30 06:09:13.098782 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:09:13.098748 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:09:14.017738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:14.017690 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" path="/var/lib/kubelet/pods/9b028d8a-67dc-41fa-ba09-b4c59b3d7777/volumes" Oct 30 06:09:14.142738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:14.142702 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt"] Oct 30 06:09:14.143027 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:14.142921 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" containerID="cri-o://86b685673b0623af42f2f3fdedd192f7c73cdfa9634f2c589c322b87c09b432a" gracePeriod=30 Oct 30 06:09:16.380729 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:16.380690 5980 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:16.381151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:16.380755 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:19.426142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.423964 5980 generic.go:356] "Generic (PLEG): container finished" podID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerID="86b685673b0623af42f2f3fdedd192f7c73cdfa9634f2c589c322b87c09b432a" exitCode=0 Oct 30 06:09:19.426142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.424083 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" event={"ID":"e5d66026-513e-4a09-bf34-e11a07bec5aa","Type":"ContainerDied","Data":"86b685673b0623af42f2f3fdedd192f7c73cdfa9634f2c589c322b87c09b432a"} Oct 30 06:09:19.520404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.520093 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:09:19.604022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.603967 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604044 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604206 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604246 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604285 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604381 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604434 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604614 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.604685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.604673 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo\") pod \"e5d66026-513e-4a09-bf34-e11a07bec5aa\" (UID: \"e5d66026-513e-4a09-bf34-e11a07bec5aa\") " Oct 30 06:09:19.605447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.605418 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.605939 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.605919 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:19.606264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.606241 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.606459 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.606434 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.607508 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.607478 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.607876 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.607852 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.607961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.607884 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.607961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.607901 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.607961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.607913 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5d66026-513e-4a09-bf34-e11a07bec5aa-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.607961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.607926 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.615675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.615607 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.615675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.615626 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:19.616097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.616073 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "kube-api-access-jh8xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:19.617547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.617518 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo" pod "e5d66026-513e-4a09-bf34-e11a07bec5aa" (UID: "e5d66026-513e-4a09-bf34-e11a07bec5aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:19.710417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.709898 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh8xs\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-kube-api-access-jh8xs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.710417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.709952 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.710417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.709967 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.710417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:19.709978 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5d66026-513e-4a09-bf34-e11a07bec5aa-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:20.433268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:20.433240 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" Oct 30 06:09:20.433737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:20.433269 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt" event={"ID":"e5d66026-513e-4a09-bf34-e11a07bec5aa","Type":"ContainerDied","Data":"ad0c71f6fe97c9d32e92b4d1caab098098372971275552cd2d395b0e2ebde0df"} Oct 30 06:09:20.433737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:20.433336 5980 scope.go:122] "RemoveContainer" containerID="86b685673b0623af42f2f3fdedd192f7c73cdfa9634f2c589c322b87c09b432a" Oct 30 06:09:20.507260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:20.507221 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt"] Oct 30 06:09:20.519972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:20.519933 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-855wt"] Oct 30 06:09:22.018647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:22.018607 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" path="/var/lib/kubelet/pods/e5d66026-513e-4a09-bf34-e11a07bec5aa/volumes" Oct 30 06:09:29.461370 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:29.461335 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4"] Oct 30 06:09:29.461935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:29.461541 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" containerID="cri-o://03249d468a46d7b6c5ded63a4c167799916567f37b8eca6a7f873f4e36d6ac19" gracePeriod=30 Oct 30 06:09:29.850976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:29.850943 5980 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:29.851202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:29.851008 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:34.709339 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.709301 5980 generic.go:356] "Generic (PLEG): container finished" podID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerID="03249d468a46d7b6c5ded63a4c167799916567f37b8eca6a7f873f4e36d6ac19" exitCode=0 Oct 30 06:09:34.709896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.709378 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" event={"ID":"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f","Type":"ContainerDied","Data":"03249d468a46d7b6c5ded63a4c167799916567f37b8eca6a7f873f4e36d6ac19"} Oct 30 06:09:34.710003 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.709965 5980 scope.go:122] "RemoveContainer" containerID="39ed0ad813770ba42c775c05df852a1c62e3a6e5bba9c4eef9fa1d0840daa8ad" Oct 30 06:09:34.782185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.782152 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:09:34.895964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.895912 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896057 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896152 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896203 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896243 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896306 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896510 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896480 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896562 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.896656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896639 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token\") pod \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\" (UID: \"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f\") " Oct 30 06:09:34.897463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896301 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.897463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896627 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:34.897463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.896896 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.897705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.897605 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.897705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.897629 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.897705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.897644 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.898334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.898303 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.898587 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.898568 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.901356 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.901318 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:34.901688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.901659 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.902327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.902296 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "kube-api-access-z9c2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.902884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.902856 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy" pod "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" (UID: "a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.999317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.999228 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9c2j\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-kube-api-access-z9c2j\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.999317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.999259 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.999317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.999269 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.999317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.999278 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.999317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.999287 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.999317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:34.999295 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:09:35.718681 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:35.718655 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" Oct 30 06:09:35.719133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:35.718646 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4" event={"ID":"a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f","Type":"ContainerDied","Data":"aeb53b8dbb435988a3468b7558095078dce2b4a0942707179a4689a0a83b5c91"} Oct 30 06:09:35.719133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:35.718785 5980 scope.go:122] "RemoveContainer" containerID="03249d468a46d7b6c5ded63a4c167799916567f37b8eca6a7f873f4e36d6ac19" Oct 30 06:09:35.816582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:35.816546 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4"] Oct 30 06:09:35.826717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:35.826684 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-bmvs4"] Oct 30 06:09:36.037253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:36.036950 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" path="/var/lib/kubelet/pods/a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f/volumes" Oct 30 06:09:43.092249 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:09:43.092206 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:09:52.797600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.797556 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf"] Oct 30 06:09:52.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798515 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" Oct 30 06:09:52.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798540 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798563 5980 cpu_manager.go:414] "containerMap: removing container" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerName="payload-pre-processing" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798573 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerName="payload-pre-processing" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798586 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798593 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798605 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798612 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798632 5980 cpu_manager.go:414] "containerMap: removing container" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerName="main" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798639 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerName="main" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798650 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ef86243d-038b-45e6-95ee-aa4a2f279bde" containerName="payload-processing" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798658 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef86243d-038b-45e6-95ee-aa4a2f279bde" containerName="payload-processing" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798685 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a2d9f949-38b1-4570-a219-5cc5649f5e4f" containerName="maas-api" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798693 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d9f949-38b1-4570-a219-5cc5649f5e4f" containerName="maas-api" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798708 5980 cpu_manager.go:414] "containerMap: removing container" podUID="cbdf8e63-5588-49de-a008-b864cd633447" containerName="payload-pre-processing" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798717 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdf8e63-5588-49de-a008-b864cd633447" containerName="payload-pre-processing" Oct 30 06:09:52.798727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798728 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798735 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798763 5980 cpu_manager.go:414] "containerMap: removing container" podUID="debeaae1-7065-4605-bc17-0cf35365f795" containerName="maas-api" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798771 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debeaae1-7065-4605-bc17-0cf35365f795" containerName="maas-api" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798797 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798804 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798979 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d9f949-38b1-4570-a219-5cc5649f5e4f" containerName="maas-api" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.798993 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdf8e63-5588-49de-a008-b864cd633447" containerName="payload-pre-processing" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799005 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799016 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="debeaae1-7065-4605-bc17-0cf35365f795" containerName="maas-api" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799027 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c203a63b-accb-4462-88a0-1b3dc81fa87b" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799038 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799052 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="968c465c-a936-4ca0-acd4-10c7aa3dc2fd" containerName="payload-pre-processing" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799062 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5d66026-513e-4a09-bf34-e11a07bec5aa" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799071 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799082 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fa82d8-d669-4aeb-acab-6f9714712db5" containerName="main" Oct 30 06:09:52.799467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.799092 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef86243d-038b-45e6-95ee-aa4a2f279bde" containerName="payload-processing" Oct 30 06:09:52.803451 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.803412 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.817389 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.817357 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.817607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.817583 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.817678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.817644 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.817763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.817747 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.817951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.817921 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.818084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.818067 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.818169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.818140 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.818235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.818183 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.818293 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.818243 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllwd\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.826092 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.826059 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf"] Oct 30 06:09:52.835202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.835171 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d3e7e6-openshift-default-dockercfg-vchrg\"" Oct 30 06:09:52.919263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919224 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919366 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919407 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919455 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919507 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919582 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllwd\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919600 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919665 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919731 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919775 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919821 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.919950 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.919900 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.920184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.920161 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.920312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.920207 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.922970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.922944 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.923215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.923191 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.947096 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.947050 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:52.951201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:52.951159 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllwd\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:53.121672 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.121619 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:53.509841 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:09:53.508935 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a388c82_7c52_47fc_bda2_f77c7a3c60c2.slice/crio-1db78c4d51c3c17257ee3201d354e42474c826a65f7dbe5b52f081077f395109 WatchSource:0}: Error finding container 1db78c4d51c3c17257ee3201d354e42474c826a65f7dbe5b52f081077f395109: Status 404 returned error can't find the container with id 1db78c4d51c3c17257ee3201d354e42474c826a65f7dbe5b52f081077f395109 Oct 30 06:09:53.510629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.510601 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf"] Oct 30 06:09:53.512907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.512861 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:53.512998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.512959 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:53.513043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.513000 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:53.908925 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.908887 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" event={"ID":"0a388c82-7c52-47fc-bda2-f77c7a3c60c2","Type":"ContainerStarted","Data":"c7afd97fa2d3abac641ca69114f303a38772c77aac95d415eca8d319176c57a6"} Oct 30 06:09:53.909363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.908950 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" event={"ID":"0a388c82-7c52-47fc-bda2-f77c7a3c60c2","Type":"ContainerStarted","Data":"1db78c4d51c3c17257ee3201d354e42474c826a65f7dbe5b52f081077f395109"} Oct 30 06:09:53.963296 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:53.963223 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podStartSLOduration=1.963205237 podStartE2EDuration="1.963205237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:53.962707766 +0000 UTC m=+2248.070746800" watchObservedRunningTime="2026-10-30 06:09:53.963205237 +0000 UTC m=+2248.071244287" Oct 30 06:09:54.122706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:54.122662 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:09:54.124541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:54.124510 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:09:54.124702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:54.124568 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:09:55.123201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:55.122790 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:09:55.123201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:55.122867 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:09:56.122435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:56.122395 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:09:56.122653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:56.122461 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:09:57.122353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:57.122316 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:09:57.122761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:57.122377 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:09:58.132147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:58.131244 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:09:58.132147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:58.131309 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:09:59.122255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:59.122215 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:09:59.122448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:59.122274 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:09:59.484596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:59.476361 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7"] Oct 30 06:09:59.484596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:09:59.476618 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" containerID="cri-o://edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0" gracePeriod=30 Oct 30 06:10:00.121942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:00.121904 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 06:10:00.122185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:00.121967 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 06:10:01.229520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:01.228507 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:10:01.233074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:01.230207 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:10:01.236145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:01.235546 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:10:01.725562 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:01.725523 5980 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:01.725745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:01.725584 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:04.829284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.829253 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:10:04.919077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919029 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919248 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919308 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919492 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919474 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919524 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919575 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919627 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919756 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.919964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.919929 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket\") pod \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\" (UID: \"ef5dc528-308f-4d4b-9484-1edbaa20c8b6\") " Oct 30 06:10:04.921276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.920843 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.921773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.921751 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.922180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.922087 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.922318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.922220 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:04.922501 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.922477 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.925258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.925220 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "kube-api-access-k4qvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:04.926074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.926046 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:04.926234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.926210 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:04.926488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:04.926464 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy" pod "ef5dc528-308f-4d4b-9484-1edbaa20c8b6" (UID: "ef5dc528-308f-4d4b-9484-1edbaa20c8b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:05.022833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022793 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.022833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022826 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.022833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022834 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.022833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022843 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.023191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022850 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.023191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022858 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.023191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022869 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.023191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022879 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.023191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.022886 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qvq\" (UniqueName: \"kubernetes.io/projected/ef5dc528-308f-4d4b-9484-1edbaa20c8b6-kube-api-access-k4qvq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.131815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.131777 5980 generic.go:356] "Generic (PLEG): container finished" podID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerID="edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0" exitCode=0 Oct 30 06:10:05.132266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.132239 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" event={"ID":"ef5dc528-308f-4d4b-9484-1edbaa20c8b6","Type":"ContainerDied","Data":"edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0"} Oct 30 06:10:05.132482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.132460 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" event={"ID":"ef5dc528-308f-4d4b-9484-1edbaa20c8b6","Type":"ContainerDied","Data":"c29783c12bc904ecdf0f1d19e141a1b3444123084d29777034617950a35663be"} Oct 30 06:10:05.132647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.132602 5980 scope.go:122] "RemoveContainer" containerID="edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0" Oct 30 06:10:05.132827 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.132811 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7" Oct 30 06:10:05.294469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.293477 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7"] Oct 30 06:10:05.319427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.316277 5980 scope.go:122] "RemoveContainer" containerID="edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0" Oct 30 06:10:05.319864 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:05.319831 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0\": container with ID starting with edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0 not found: ID does not exist" containerID="edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0" Oct 30 06:10:05.320070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.320041 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0"} err="failed to get container status \"edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0\": rpc error: code = NotFound desc = could not find container \"edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0\": container with ID starting with edde439f0db388acec353533e8d45cc6b03dc18b010207da5a23fdb702cb1de0 not found: ID does not exist" Oct 30 06:10:05.337707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:05.336277 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-4rdl7"] Oct 30 06:10:06.017666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:06.017580 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" path="/var/lib/kubelet/pods/ef5dc528-308f-4d4b-9484-1edbaa20c8b6/volumes" Oct 30 06:10:13.097639 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:13.097597 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:10:15.078242 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:15.078215 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj"] Oct 30 06:10:21.277879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:21.277829 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp"] Oct 30 06:10:23.811896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:23.811852 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj"] Oct 30 06:10:23.812466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:23.812139 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" podUID="3537c2ea-3c10-4101-9af9-a84b5204c016" containerName="maas-api" containerID="cri-o://86f308c20b62f149ed2f7d27330efae81aa1c32d57d604ae168f7c5257769f83" gracePeriod=30 Oct 30 06:10:24.415798 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.415759 5980 generic.go:356] "Generic (PLEG): container finished" podID="3537c2ea-3c10-4101-9af9-a84b5204c016" containerID="86f308c20b62f149ed2f7d27330efae81aa1c32d57d604ae168f7c5257769f83" exitCode=0 Oct 30 06:10:24.416256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.416229 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" event={"ID":"3537c2ea-3c10-4101-9af9-a84b5204c016","Type":"ContainerDied","Data":"86f308c20b62f149ed2f7d27330efae81aa1c32d57d604ae168f7c5257769f83"} Oct 30 06:10:24.457074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.457048 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:10:24.637638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.637606 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh\" (UniqueName: \"kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh\") pod \"3537c2ea-3c10-4101-9af9-a84b5204c016\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " Oct 30 06:10:24.638029 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.638002 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\") pod \"3537c2ea-3c10-4101-9af9-a84b5204c016\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " Oct 30 06:10:24.638579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.638416 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca\") pod \"3537c2ea-3c10-4101-9af9-a84b5204c016\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " Oct 30 06:10:24.639649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.639630 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs\") pod \"3537c2ea-3c10-4101-9af9-a84b5204c016\" (UID: \"3537c2ea-3c10-4101-9af9-a84b5204c016\") " Oct 30 06:10:24.640220 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.639530 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca" pod "3537c2ea-3c10-4101-9af9-a84b5204c016" (UID: "3537c2ea-3c10-4101-9af9-a84b5204c016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:24.641554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.641528 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3537c2ea-3c10-4101-9af9-a84b5204c016-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:24.668084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.668035 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh"] Oct 30 06:10:24.669401 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.669363 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" containerID="cri-o://c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f" gracePeriod=30 Oct 30 06:10:24.674553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.674503 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh" pod "3537c2ea-3c10-4101-9af9-a84b5204c016" (UID: "3537c2ea-3c10-4101-9af9-a84b5204c016"). InnerVolumeSpecName "kube-api-access-vxrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:24.674728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.674709 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs" pod "3537c2ea-3c10-4101-9af9-a84b5204c016" (UID: "3537c2ea-3c10-4101-9af9-a84b5204c016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:24.674889 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.674844 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls" pod "3537c2ea-3c10-4101-9af9-a84b5204c016" (UID: "3537c2ea-3c10-4101-9af9-a84b5204c016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:24.744174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.744134 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrmh\" (UniqueName: \"kubernetes.io/projected/3537c2ea-3c10-4101-9af9-a84b5204c016-kube-api-access-vxrmh\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:24.744371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.744185 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:24.744371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.744200 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3537c2ea-3c10-4101-9af9-a84b5204c016-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:24.827645 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.827605 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf"] Oct 30 06:10:24.828156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.827936 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" containerID="cri-o://d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba" gracePeriod=30 Oct 30 06:10:24.881601 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.881137 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn"] Oct 30 06:10:24.882273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:24.882157 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" containerID="cri-o://54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8" gracePeriod=30 Oct 30 06:10:25.425910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.425874 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" event={"ID":"3537c2ea-3c10-4101-9af9-a84b5204c016","Type":"ContainerDied","Data":"9d548b185d0f58735467906f2f151f48c141dc853cbf11a353979ad91d257dcd"} Oct 30 06:10:25.426168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.425920 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj" Oct 30 06:10:25.426168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.425940 5980 scope.go:122] "RemoveContainer" containerID="86f308c20b62f149ed2f7d27330efae81aa1c32d57d604ae168f7c5257769f83" Oct 30 06:10:25.486897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.486748 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj"] Oct 30 06:10:25.490625 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.490587 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c1d09f-5847bb6558-s26vj"] Oct 30 06:10:25.663414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.663379 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp"] Oct 30 06:10:25.688416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:25.688321 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm"] Oct 30 06:10:26.029853 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.029753 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3537c2ea-3c10-4101-9af9-a84b5204c016" path="/var/lib/kubelet/pods/3537c2ea-3c10-4101-9af9-a84b5204c016/volumes" Oct 30 06:10:26.295195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.290420 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" podUID="3467707d-815b-4d39-862c-783626de50ca" containerName="payload-processing" containerID="cri-o://eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215" gracePeriod=25 Oct 30 06:10:26.750915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.750886 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:10:26.828848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.828746 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml\" (UniqueName: \"kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml\") pod \"3467707d-815b-4d39-862c-783626de50ca\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " Oct 30 06:10:26.829023 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.828923 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume\") pod \"3467707d-815b-4d39-862c-783626de50ca\" (UID: \"3467707d-815b-4d39-862c-783626de50ca\") " Oct 30 06:10:26.829358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.829313 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume" pod "3467707d-815b-4d39-862c-783626de50ca" (UID: "3467707d-815b-4d39-862c-783626de50ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:26.829830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.829805 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3467707d-815b-4d39-862c-783626de50ca-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.831477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.831446 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml" pod "3467707d-815b-4d39-862c-783626de50ca" (UID: "3467707d-815b-4d39-862c-783626de50ca"). InnerVolumeSpecName "kube-api-access-qdqml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:26.931524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:26.931480 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqml\" (UniqueName: \"kubernetes.io/projected/3467707d-815b-4d39-862c-783626de50ca-kube-api-access-qdqml\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:27.457547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.457514 5980 generic.go:356] "Generic (PLEG): container finished" podID="3467707d-815b-4d39-862c-783626de50ca" containerID="eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215" exitCode=0 Oct 30 06:10:27.458064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.457553 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" Oct 30 06:10:27.458064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.457598 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" event={"ID":"3467707d-815b-4d39-862c-783626de50ca","Type":"ContainerDied","Data":"eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215"} Oct 30 06:10:27.458064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.457659 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp" event={"ID":"3467707d-815b-4d39-862c-783626de50ca","Type":"ContainerDied","Data":"96cb05eea051206a78eb1b375e4521ecb4d97b7dff2d0c9d16e61ae8580fb919"} Oct 30 06:10:27.458064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.457692 5980 scope.go:122] "RemoveContainer" containerID="eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215" Oct 30 06:10:27.476188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.476084 5980 scope.go:122] "RemoveContainer" containerID="eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215" Oct 30 06:10:27.476479 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:27.476454 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215\": container with ID starting with eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215 not found: ID does not exist" containerID="eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215" Oct 30 06:10:27.476547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.476494 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215"} err="failed to get container status \"eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215\": rpc error: code = NotFound desc = could not find container \"eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215\": container with ID starting with eb9260e675216aba2d222304ec7fa7a20eec6ae280e1252c9021e84818929215 not found: ID does not exist" Oct 30 06:10:27.515449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.515410 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp"] Oct 30 06:10:27.528520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:27.528479 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c1d09f-7f6b89c47-djdjp"] Oct 30 06:10:28.016040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:28.016000 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3467707d-815b-4d39-862c-783626de50ca" path="/var/lib/kubelet/pods/3467707d-815b-4d39-862c-783626de50ca/volumes" Oct 30 06:10:30.103170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.103143 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 06:10:30.127257 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.127173 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" podUID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" containerName="main" containerID="cri-o://2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b" gracePeriod=45 Oct 30 06:10:30.190576 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190532 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.190777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190630 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.190847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190787 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.190847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190841 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.190960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190897 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.190960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190926 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.191059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.190959 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.191227 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.191198 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.191305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.191289 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert\") pod \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\" (UID: \"936d2db7-98ed-4b19-bf3c-d302b57cf15e\") " Oct 30 06:10:30.193740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.192358 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.193740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.192528 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.193740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.192808 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.193740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.193133 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:30.193740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.193549 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.194812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.194782 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.194891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.194814 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.194891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.194828 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.194891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.194842 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.194891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.194857 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.199328 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.199290 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:30.205419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.205380 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "kube-api-access-2d2qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.205419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.205407 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.206414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.206358 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy" pod "936d2db7-98ed-4b19-bf3c-d302b57cf15e" (UID: "936d2db7-98ed-4b19-bf3c-d302b57cf15e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.223060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.223010 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" podUID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": dial tcp 10.134.0.129:8000: connect: connection refused" Oct 30 06:10:30.253746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.253714 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 06:10:30.282702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.282665 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 06:10:30.296952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.296906 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.296952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.296946 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.296952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.296960 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2qr\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-kube-api-access-2d2qr\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.297347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.296974 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936d2db7-98ed-4b19-bf3c-d302b57cf15e-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.398131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.397945 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.398131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.398055 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.399083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.398885 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.399083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.398097 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399065 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399134 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.399390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399168 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399254 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.399390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399297 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.399390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399328 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399379 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399446 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399496 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.399712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399572 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.399712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399645 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.399712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399680 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.400011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399738 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.400011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399790 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket\") pod \"66025822-a5de-4e85-8d69-b7fb524b5858\" (UID: \"66025822-a5de-4e85-8d69-b7fb524b5858\") " Oct 30 06:10:30.400011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.399895 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs\") pod \"9196a232-a912-4924-9b88-05f14a101fad\" (UID: \"9196a232-a912-4924-9b88-05f14a101fad\") " Oct 30 06:10:30.400522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.400491 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.401429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.401394 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.401902 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.401875 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.403382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.402398 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.403382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.402777 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.403382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.403100 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.403382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.403297 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.404080 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404050 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404085 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404100 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404148 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404161 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404174 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404186 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404200 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.404560 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404518 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:30.404890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.404866 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:30.405168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.405147 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.408944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.408830 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "kube-api-access-dlbkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.409296 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.409271 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:30.409839 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.409810 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.410125 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.410075 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "kube-api-access-ql9zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.410446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.410415 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token" pod "66025822-a5de-4e85-8d69-b7fb524b5858" (UID: "66025822-a5de-4e85-8d69-b7fb524b5858"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.410723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.410701 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.410800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.410781 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo" pod "9196a232-a912-4924-9b88-05f14a101fad" (UID: "9196a232-a912-4924-9b88-05f14a101fad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:30.478842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.478805 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:10:30.486893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.486858 5980 generic.go:356] "Generic (PLEG): container finished" podID="66025822-a5de-4e85-8d69-b7fb524b5858" containerID="d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba" exitCode=0 Oct 30 06:10:30.487101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.486919 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" Oct 30 06:10:30.487101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.486940 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" event={"ID":"66025822-a5de-4e85-8d69-b7fb524b5858","Type":"ContainerDied","Data":"d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba"} Oct 30 06:10:30.487101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.487001 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf" event={"ID":"66025822-a5de-4e85-8d69-b7fb524b5858","Type":"ContainerDied","Data":"d9a99b4b7d22ddb615849e8530993c9e42851d4863f44cf1c059181d974febb8"} Oct 30 06:10:30.487101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.487032 5980 scope.go:122] "RemoveContainer" containerID="d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba" Oct 30 06:10:30.491479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.491314 5980 generic.go:356] "Generic (PLEG): container finished" podID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerID="c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f" exitCode=0 Oct 30 06:10:30.491479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.491397 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" Oct 30 06:10:30.491479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.491467 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" event={"ID":"936d2db7-98ed-4b19-bf3c-d302b57cf15e","Type":"ContainerDied","Data":"c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f"} Oct 30 06:10:30.491479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.491508 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh" event={"ID":"936d2db7-98ed-4b19-bf3c-d302b57cf15e","Type":"ContainerDied","Data":"cbf3d5859ddf5f1780143d77c2577ce62d79029cf28f07fe4066fdf2c749e305"} Oct 30 06:10:30.498970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.498794 5980 generic.go:356] "Generic (PLEG): container finished" podID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" containerID="2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b" exitCode=0 Oct 30 06:10:30.498970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.498862 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" Oct 30 06:10:30.499481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.498891 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" event={"ID":"7ec9e425-dacf-4638-aa69-7abfff7f0aa0","Type":"ContainerDied","Data":"2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b"} Oct 30 06:10:30.499613 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.499499 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj" event={"ID":"7ec9e425-dacf-4638-aa69-7abfff7f0aa0","Type":"ContainerDied","Data":"6c66879ba66c5253d64550e65b8dbf9b4433a76ac3c698881f0c922089e50139"} Oct 30 06:10:30.500906 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.500877 5980 generic.go:356] "Generic (PLEG): container finished" podID="9196a232-a912-4924-9b88-05f14a101fad" containerID="54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8" exitCode=0 Oct 30 06:10:30.501054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.500928 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" Oct 30 06:10:30.501054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.500951 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" event={"ID":"9196a232-a912-4924-9b88-05f14a101fad","Type":"ContainerDied","Data":"54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8"} Oct 30 06:10:30.501054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.500994 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn" event={"ID":"9196a232-a912-4924-9b88-05f14a101fad","Type":"ContainerDied","Data":"72954fe3468d7343331a5cc4a8e36704aa550df4e8938813332095b86aca077f"} Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505892 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196a232-a912-4924-9b88-05f14a101fad-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505922 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9zl\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-kube-api-access-ql9zl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505936 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505949 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66025822-a5de-4e85-8d69-b7fb524b5858-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505961 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbkl\" (UniqueName: \"kubernetes.io/projected/66025822-a5de-4e85-8d69-b7fb524b5858-kube-api-access-dlbkl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505975 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196a232-a912-4924-9b88-05f14a101fad-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505988 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196a232-a912-4924-9b88-05f14a101fad-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.505999 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66025822-a5de-4e85-8d69-b7fb524b5858-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.506012 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66025822-a5de-4e85-8d69-b7fb524b5858-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.506290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.506024 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196a232-a912-4924-9b88-05f14a101fad-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.511336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.511308 5980 scope.go:122] "RemoveContainer" containerID="5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217" Oct 30 06:10:30.526356 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.526223 5980 scope.go:122] "RemoveContainer" containerID="d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba" Oct 30 06:10:30.540630 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:30.540597 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba\": container with ID starting with d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba not found: ID does not exist" containerID="d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba" Oct 30 06:10:30.540836 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.540637 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba"} err="failed to get container status \"d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba\": rpc error: code = NotFound desc = could not find container \"d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba\": container with ID starting with d253f082cda741dcc29018d7818c4618e033b41c1d09b8bf3983cbb350acc7ba not found: ID does not exist" Oct 30 06:10:30.540836 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.540663 5980 scope.go:122] "RemoveContainer" containerID="5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217" Oct 30 06:10:30.541067 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:30.541043 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217\": container with ID starting with 5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217 not found: ID does not exist" containerID="5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217" Oct 30 06:10:30.541166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.541080 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217"} err="failed to get container status \"5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217\": rpc error: code = NotFound desc = could not find container \"5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217\": container with ID starting with 5f2829ae9673b4eb4de47784b3a9dbfb0e1c84fb0b7249dc645dda9d87dcf217 not found: ID does not exist" Oct 30 06:10:30.541166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.541126 5980 scope.go:122] "RemoveContainer" containerID="c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f" Oct 30 06:10:30.556245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.556222 5980 scope.go:122] "RemoveContainer" containerID="fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe" Oct 30 06:10:30.570279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.570231 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf"] Oct 30 06:10:30.572363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.572343 5980 scope.go:122] "RemoveContainer" containerID="c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f" Oct 30 06:10:30.572734 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:30.572714 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f\": container with ID starting with c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f not found: ID does not exist" containerID="c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f" Oct 30 06:10:30.572813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.572744 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f"} err="failed to get container status \"c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f\": rpc error: code = NotFound desc = could not find container \"c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f\": container with ID starting with c2700e6a0562e8dc34daa6721f9cbefd210f91b6c11e53d252511e86ea8a557f not found: ID does not exist" Oct 30 06:10:30.572813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.572765 5980 scope.go:122] "RemoveContainer" containerID="fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe" Oct 30 06:10:30.573079 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:30.573061 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe\": container with ID starting with fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe not found: ID does not exist" containerID="fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe" Oct 30 06:10:30.573196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.573085 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe"} err="failed to get container status \"fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe\": rpc error: code = NotFound desc = could not find container \"fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe\": container with ID starting with fd05261d9c6a4b7ec557a2b81a552c7d28d7d9f16b480e832a67c78d3388cbfe not found: ID does not exist" Oct 30 06:10:30.573196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.573101 5980 scope.go:122] "RemoveContainer" containerID="2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b" Oct 30 06:10:30.574736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.574716 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-42xrf"] Oct 30 06:10:30.594853 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.594703 5980 scope.go:122] "RemoveContainer" containerID="2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b" Oct 30 06:10:30.595302 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:30.595275 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b\": container with ID starting with 2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b not found: ID does not exist" containerID="2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b" Oct 30 06:10:30.595437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.595318 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b"} err="failed to get container status \"2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b\": rpc error: code = NotFound desc = could not find container \"2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b\": container with ID starting with 2e77a91f642faa4522a6276b8758445192e17732cd6bb917d303fd2f6d869b9b not found: ID does not exist" Oct 30 06:10:30.595437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.595347 5980 scope.go:122] "RemoveContainer" containerID="54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8" Oct 30 06:10:30.612384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.612343 5980 scope.go:122] "RemoveContainer" containerID="54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8" Oct 30 06:10:30.612844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.612816 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz\" (UniqueName: \"kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz\") pod \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " Oct 30 06:10:30.612943 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:30.612837 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8\": container with ID starting with 54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8 not found: ID does not exist" containerID="54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8" Oct 30 06:10:30.612943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.612867 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache\") pod \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " Oct 30 06:10:30.612943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.612876 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8"} err="failed to get container status \"54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8\": rpc error: code = NotFound desc = could not find container \"54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8\": container with ID starting with 54dc019275649d18e51471b72620e232ae20c15a4e867d95955e7dcad8c9f8f8 not found: ID does not exist" Oct 30 06:10:30.612943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.612903 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs\") pod \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " Oct 30 06:10:30.613084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.612990 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home\") pod \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " Oct 30 06:10:30.613343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.613187 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm\") pod \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " Oct 30 06:10:30.613343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.613226 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir\") pod \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\" (UID: \"7ec9e425-dacf-4638-aa69-7abfff7f0aa0\") " Oct 30 06:10:30.613634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.613581 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache" pod "7ec9e425-dacf-4638-aa69-7abfff7f0aa0" (UID: "7ec9e425-dacf-4638-aa69-7abfff7f0aa0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.613830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.613806 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir" pod "7ec9e425-dacf-4638-aa69-7abfff7f0aa0" (UID: "7ec9e425-dacf-4638-aa69-7abfff7f0aa0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.614219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.614195 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.614346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.614221 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.614346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.614272 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home" pod "7ec9e425-dacf-4638-aa69-7abfff7f0aa0" (UID: "7ec9e425-dacf-4638-aa69-7abfff7f0aa0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.616189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.616161 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs" pod "7ec9e425-dacf-4638-aa69-7abfff7f0aa0" (UID: "7ec9e425-dacf-4638-aa69-7abfff7f0aa0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:30.616308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.616280 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz" pod "7ec9e425-dacf-4638-aa69-7abfff7f0aa0" (UID: "7ec9e425-dacf-4638-aa69-7abfff7f0aa0"). InnerVolumeSpecName "kube-api-access-2fnqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:30.616727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.616701 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm" pod "7ec9e425-dacf-4638-aa69-7abfff7f0aa0" (UID: "7ec9e425-dacf-4638-aa69-7abfff7f0aa0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:30.618471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.618441 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh"] Oct 30 06:10:30.622204 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.622170 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-mx8xh"] Oct 30 06:10:30.654180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.654137 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn"] Oct 30 06:10:30.659315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.659278 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-rmkwn"] Oct 30 06:10:30.664412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.664365 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" containerName="payload-processing" containerID="cri-o://c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3" gracePeriod=25 Oct 30 06:10:30.688737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.688691 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" containerName="payload-pre-processing" containerID="cri-o://eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391" gracePeriod=25 Oct 30 06:10:30.715696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.715651 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnqz\" (UniqueName: \"kubernetes.io/projected/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-kube-api-access-2fnqz\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.715696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.715687 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.715923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.715706 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.715923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.715717 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ec9e425-dacf-4638-aa69-7abfff7f0aa0-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:30.849612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.849569 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj"] Oct 30 06:10:30.853085 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:30.853040 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0d64c9/test-model-0d64c9-kserve-c747557bc-zdhlj"] Oct 30 06:10:32.020787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.020735 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" path="/var/lib/kubelet/pods/66025822-a5de-4e85-8d69-b7fb524b5858/volumes" Oct 30 06:10:32.021803 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.021773 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" path="/var/lib/kubelet/pods/7ec9e425-dacf-4638-aa69-7abfff7f0aa0/volumes" Oct 30 06:10:32.022813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.022789 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9196a232-a912-4924-9b88-05f14a101fad" path="/var/lib/kubelet/pods/9196a232-a912-4924-9b88-05f14a101fad/volumes" Oct 30 06:10:32.024066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.024039 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" path="/var/lib/kubelet/pods/936d2db7-98ed-4b19-bf3c-d302b57cf15e/volumes" Oct 30 06:10:32.101525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.100976 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4"] Oct 30 06:10:32.101810 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.101279 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" containerID="cri-o://3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5" gracePeriod=30 Oct 30 06:10:32.655610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.655567 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:32.655610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:32.655584 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:37.419151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.419101 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:10:37.561965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.561929 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.561965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.561968 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.562244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562093 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.562244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562151 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.562244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562189 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562436 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562506 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562594 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562599 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562647 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562707 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.562801 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data\") pod \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\" (UID: \"e1d9e535-6153-49bb-8b17-830f60a9a3ec\") " Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563056 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.563090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563083 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.563812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563733 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.563812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563757 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.563812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563771 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.563812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563783 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.563812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.563797 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.566681 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.566648 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:37.566833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.566818 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.566951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.566933 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:37.568690 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.568661 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr" pod "e1d9e535-6153-49bb-8b17-830f60a9a3ec" (UID: "e1d9e535-6153-49bb-8b17-830f60a9a3ec"). InnerVolumeSpecName "kube-api-access-zv2qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:37.602055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.602016 5980 generic.go:356] "Generic (PLEG): container finished" podID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerID="3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5" exitCode=0 Oct 30 06:10:37.602055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.602068 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" Oct 30 06:10:37.602366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.602092 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" event={"ID":"e1d9e535-6153-49bb-8b17-830f60a9a3ec","Type":"ContainerDied","Data":"3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5"} Oct 30 06:10:37.602366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.602188 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4" event={"ID":"e1d9e535-6153-49bb-8b17-830f60a9a3ec","Type":"ContainerDied","Data":"5bf925e3aa9aec080d506f547576f11c092b46a9004bb9ef024f9c85076c7b27"} Oct 30 06:10:37.602366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.602211 5980 scope.go:122] "RemoveContainer" containerID="3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5" Oct 30 06:10:37.616188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.616162 5980 scope.go:122] "RemoveContainer" containerID="3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5" Oct 30 06:10:37.616545 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:37.616519 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5\": container with ID starting with 3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5 not found: ID does not exist" containerID="3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5" Oct 30 06:10:37.616642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.616553 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5"} err="failed to get container status \"3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5\": rpc error: code = NotFound desc = could not find container \"3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5\": container with ID starting with 3bc2a9808e07c9ccdeecf19a255f03812d6eaabcb4b5326ce125642d81ca25d5 not found: ID does not exist" Oct 30 06:10:37.660515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.660421 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4"] Oct 30 06:10:37.665128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.665083 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.665128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.665123 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv2qr\" (UniqueName: \"kubernetes.io/projected/e1d9e535-6153-49bb-8b17-830f60a9a3ec-kube-api-access-zv2qr\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.665372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.665140 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.665372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.665155 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1d9e535-6153-49bb-8b17-830f60a9a3ec-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.666675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:37.666648 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-h7sw4"] Oct 30 06:10:38.016395 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:38.016310 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" path="/var/lib/kubelet/pods/e1d9e535-6153-49bb-8b17-830f60a9a3ec/volumes" Oct 30 06:10:42.654503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:42.654460 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:42.654941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:42.654538 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:43.104976 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:43.104939 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:10:43.318631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:43.318594 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb"] Oct 30 06:10:43.352640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:43.352602 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln"] Oct 30 06:10:48.319678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:48.319627 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerName="payload-processing" containerID="cri-o://6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14" gracePeriod=25 Oct 30 06:10:48.400424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:48.391500 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerName="payload-pre-processing" containerID="cri-o://a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e" gracePeriod=25 Oct 30 06:10:51.053898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.053859 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:10:51.080362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.080324 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:10:51.155281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.155249 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx\" (UniqueName: \"kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx\") pod \"e628adbf-9199-4539-8bed-1f4e186922e3\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " Oct 30 06:10:51.155503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.155390 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n\" (UniqueName: \"kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n\") pod \"b298cb49-585f-46f9-be12-67cd92b13c06\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " Oct 30 06:10:51.155503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.155434 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume\") pod \"b298cb49-585f-46f9-be12-67cd92b13c06\" (UID: \"b298cb49-585f-46f9-be12-67cd92b13c06\") " Oct 30 06:10:51.155632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.155618 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume\") pod \"e628adbf-9199-4539-8bed-1f4e186922e3\" (UID: \"e628adbf-9199-4539-8bed-1f4e186922e3\") " Oct 30 06:10:51.156072 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.156046 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume" pod "b298cb49-585f-46f9-be12-67cd92b13c06" (UID: "b298cb49-585f-46f9-be12-67cd92b13c06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:51.156545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.156513 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume" pod "e628adbf-9199-4539-8bed-1f4e186922e3" (UID: "e628adbf-9199-4539-8bed-1f4e186922e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:51.157052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.157030 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b298cb49-585f-46f9-be12-67cd92b13c06-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.158564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.158502 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n" pod "b298cb49-585f-46f9-be12-67cd92b13c06" (UID: "b298cb49-585f-46f9-be12-67cd92b13c06"). InnerVolumeSpecName "kube-api-access-5st4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:51.158729 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.158614 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx" pod "e628adbf-9199-4539-8bed-1f4e186922e3" (UID: "e628adbf-9199-4539-8bed-1f4e186922e3"). InnerVolumeSpecName "kube-api-access-c75dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:51.268192 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.264625 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e628adbf-9199-4539-8bed-1f4e186922e3-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.268192 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.264673 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75dx\" (UniqueName: \"kubernetes.io/projected/e628adbf-9199-4539-8bed-1f4e186922e3-kube-api-access-c75dx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.268192 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.264688 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5st4n\" (UniqueName: \"kubernetes.io/projected/b298cb49-585f-46f9-be12-67cd92b13c06-kube-api-access-5st4n\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.726844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.726806 5980 generic.go:356] "Generic (PLEG): container finished" podID="b298cb49-585f-46f9-be12-67cd92b13c06" containerID="eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391" exitCode=0 Oct 30 06:10:51.727028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.726861 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" Oct 30 06:10:51.727028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.726899 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" event={"ID":"b298cb49-585f-46f9-be12-67cd92b13c06","Type":"ContainerDied","Data":"eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391"} Oct 30 06:10:51.727028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.726963 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm" event={"ID":"b298cb49-585f-46f9-be12-67cd92b13c06","Type":"ContainerDied","Data":"1d6f166903579a9d3cddc46aafff4e41cd80529e3655074ec316031e983a20fb"} Oct 30 06:10:51.727028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.726999 5980 scope.go:122] "RemoveContainer" containerID="eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391" Oct 30 06:10:51.728656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.728632 5980 generic.go:356] "Generic (PLEG): container finished" podID="e628adbf-9199-4539-8bed-1f4e186922e3" containerID="c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3" exitCode=0 Oct 30 06:10:51.728788 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.728691 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" Oct 30 06:10:51.728788 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.728726 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" event={"ID":"e628adbf-9199-4539-8bed-1f4e186922e3","Type":"ContainerDied","Data":"c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3"} Oct 30 06:10:51.728890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.728795 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp" event={"ID":"e628adbf-9199-4539-8bed-1f4e186922e3","Type":"ContainerDied","Data":"c9ecb630eb933dd85326026f62d9a1ad4c196252a24b3c497070f8d9c62de550"} Oct 30 06:10:51.739986 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.739966 5980 scope.go:122] "RemoveContainer" containerID="eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391" Oct 30 06:10:51.740408 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:51.740389 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391\": container with ID starting with eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391 not found: ID does not exist" containerID="eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391" Oct 30 06:10:51.740486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.740416 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391"} err="failed to get container status \"eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391\": rpc error: code = NotFound desc = could not find container \"eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391\": container with ID starting with eaef29eabf07fc6ee989508bb77e5694793bae221284a18ad70d6d11c2225391 not found: ID does not exist" Oct 30 06:10:51.740486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.740434 5980 scope.go:122] "RemoveContainer" containerID="c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3" Oct 30 06:10:51.752497 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.752475 5980 scope.go:122] "RemoveContainer" containerID="c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3" Oct 30 06:10:51.752819 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:10:51.752800 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3\": container with ID starting with c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3 not found: ID does not exist" containerID="c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3" Oct 30 06:10:51.752869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.752832 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3"} err="failed to get container status \"c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3\": rpc error: code = NotFound desc = could not find container \"c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3\": container with ID starting with c068319926d202065899c7fe147cef5fd9f9834954c04e47d45077d6ba6c1ae3 not found: ID does not exist" Oct 30 06:10:51.783794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.783753 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp"] Oct 30 06:10:51.798780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.798733 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7df9fa-cc7758555-rrcvp"] Oct 30 06:10:51.825175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.825098 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm"] Oct 30 06:10:51.828351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:51.828326 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7df9fa-7485cccd6c-mxttm"] Oct 30 06:10:52.016464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:52.016368 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" path="/var/lib/kubelet/pods/b298cb49-585f-46f9-be12-67cd92b13c06/volumes" Oct 30 06:10:52.016966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:52.016942 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" path="/var/lib/kubelet/pods/e628adbf-9199-4539-8bed-1f4e186922e3/volumes" Oct 30 06:10:52.648131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:52.644257 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:53.633171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:53.633127 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:55.837138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:55.836320 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz"] Oct 30 06:10:55.837138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:55.836597 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" containerID="cri-o://22c0c7a8c47f8693bba21556cfd43582aa679b4a4022abe7f033d52f2c7cd610" gracePeriod=30 Oct 30 06:10:57.795862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.795805 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg"] Oct 30 06:10:57.796863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.796834 5980 cpu_manager.go:414] "containerMap: removing container" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" Oct 30 06:10:57.796863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.796861 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" Oct 30 06:10:57.796863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.796896 5980 cpu_manager.go:414] "containerMap: removing container" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" Oct 30 06:10:57.796863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.796904 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.796978 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.796989 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797005 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797012 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797030 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3537c2ea-3c10-4101-9af9-a84b5204c016" containerName="maas-api" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797036 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3537c2ea-3c10-4101-9af9-a84b5204c016" containerName="maas-api" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797058 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" containerName="payload-processing" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797066 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" containerName="payload-processing" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797084 5980 cpu_manager.go:414] "containerMap: removing container" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797090 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797099 5980 cpu_manager.go:414] "containerMap: removing container" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" containerName="payload-pre-processing" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797122 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" containerName="payload-pre-processing" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797133 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" containerName="main" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797140 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" containerName="main" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797155 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797164 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797177 5980 cpu_manager.go:414] "containerMap: removing container" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797184 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797197 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3467707d-815b-4d39-862c-783626de50ca" containerName="payload-processing" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797204 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3467707d-815b-4d39-862c-783626de50ca" containerName="payload-processing" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797223 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" Oct 30 06:10:57.797325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797230 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b028d8a-67dc-41fa-ba09-b4c59b3d7777" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797394 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797408 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="936d2db7-98ed-4b19-bf3c-d302b57cf15e" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797420 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11c11d6-b5a0-4b6d-a43f-67560b9b6c9f" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797433 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9196a232-a912-4924-9b88-05f14a101fad" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797440 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d9e535-6153-49bb-8b17-830f60a9a3ec" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797450 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3537c2ea-3c10-4101-9af9-a84b5204c016" containerName="maas-api" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797461 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e628adbf-9199-4539-8bed-1f4e186922e3" containerName="payload-processing" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797470 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797512 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b298cb49-585f-46f9-be12-67cd92b13c06" containerName="payload-pre-processing" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797522 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec9e425-dacf-4638-aa69-7abfff7f0aa0" containerName="main" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797532 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5dc528-308f-4d4b-9484-1edbaa20c8b6" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797540 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="66025822-a5de-4e85-8d69-b7fb524b5858" containerName="istio-proxy" Oct 30 06:10:57.798539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.797548 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3467707d-815b-4d39-862c-783626de50ca" containerName="payload-processing" Oct 30 06:10:57.804996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.804961 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.841377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.841339 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg"] Oct 30 06:10:57.850277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.850246 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-91f42c-openshift-default-dockercfg-5blpg\"" Oct 30 06:10:57.903895 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.903844 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwq92\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.904244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.904224 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.904449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.904429 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.904628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.904610 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.904966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.904937 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.905186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.905160 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.905357 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.905332 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.905519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.905493 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:57.905643 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:57.905628 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.074967 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075074 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075152 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075200 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075232 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075302 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwq92\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075361 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075429 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.075489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.075495 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.078177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.078147 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.078797 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.078766 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.079536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.079514 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.080016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.079994 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.086847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.080211 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.103176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.102474 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.106754 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.106709 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.134020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.133619 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwq92\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.169771 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.169693 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.423626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.423540 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:58.788279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.788240 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg"] Oct 30 06:10:58.790478 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:10:58.790438 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode709aeb7_5931_4b2d_a7a1_48f2f0e1adf8.slice/crio-97a5fc3ebbb2383727f0b2c12c3839825536953d1728e70ba188f31bfae27343 WatchSource:0}: Error finding container 97a5fc3ebbb2383727f0b2c12c3839825536953d1728e70ba188f31bfae27343: Status 404 returned error can't find the container with id 97a5fc3ebbb2383727f0b2c12c3839825536953d1728e70ba188f31bfae27343 Oct 30 06:10:58.794471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.794415 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:58.794645 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.794521 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:58.794645 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.794565 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:58.939704 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.939662 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" event={"ID":"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8","Type":"ContainerStarted","Data":"c528fbd23330b3b9f3168ea8220390c3daf0b57c8d26c9721c6a24738ff7adcf"} Oct 30 06:10:58.940205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.939720 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" event={"ID":"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8","Type":"ContainerStarted","Data":"97a5fc3ebbb2383727f0b2c12c3839825536953d1728e70ba188f31bfae27343"} Oct 30 06:10:58.989948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:58.989877 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podStartSLOduration=1.9898587540000001 podStartE2EDuration="1.989858754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:58.986979141 +0000 UTC m=+2313.095018174" watchObservedRunningTime="2026-10-30 06:10:58.989858754 +0000 UTC m=+2313.097897842" Oct 30 06:10:59.424370 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:59.424333 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:59.431918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:59.431888 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:59.949010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:59.948970 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:10:59.950483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:10:59.950459 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:11:00.958533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:00.958482 5980 generic.go:356] "Generic (PLEG): container finished" podID="073b49be-33b8-4fa0-8d02-599933418225" containerID="22c0c7a8c47f8693bba21556cfd43582aa679b4a4022abe7f033d52f2c7cd610" exitCode=0 Oct 30 06:11:00.959011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:00.958553 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" event={"ID":"073b49be-33b8-4fa0-8d02-599933418225","Type":"ContainerDied","Data":"22c0c7a8c47f8693bba21556cfd43582aa679b4a4022abe7f033d52f2c7cd610"} Oct 30 06:11:00.959011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:00.958631 5980 scope.go:122] "RemoveContainer" containerID="193c4f99ced2b86f73c51a5def782b9684d50eaa5d9a64a136a60c9b45297088" Oct 30 06:11:01.132229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.132151 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:11:01.241717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.241676 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.241918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.241780 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.241918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.241842 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.242051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.241934 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.242161 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242072 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.242283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242252 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.242342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242273 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.242595 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242571 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.242728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242626 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.242728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242696 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.242856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.242817 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert\") pod \"073b49be-33b8-4fa0-8d02-599933418225\" (UID: \"073b49be-33b8-4fa0-8d02-599933418225\") " Oct 30 06:11:01.243206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.243180 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.243720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.243666 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:01.243885 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.243866 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.243993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.243981 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.244082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.244071 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.244202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.243898 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.246120 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.244944 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:01.246120 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.244989 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "kube-api-access-d29hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.246120 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.245068 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.246593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.246564 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy" pod "073b49be-33b8-4fa0-8d02-599933418225" (UID: "073b49be-33b8-4fa0-8d02-599933418225"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.345455 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.345417 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.345455 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.345454 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/073b49be-33b8-4fa0-8d02-599933418225-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.345455 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.345467 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/073b49be-33b8-4fa0-8d02-599933418225-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.345767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.345476 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29hn\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-kube-api-access-d29hn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.345767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.345484 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/073b49be-33b8-4fa0-8d02-599933418225-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.345767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.345493 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/073b49be-33b8-4fa0-8d02-599933418225-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.969648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.969617 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" Oct 30 06:11:01.970164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.969648 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz" event={"ID":"073b49be-33b8-4fa0-8d02-599933418225","Type":"ContainerDied","Data":"ed53f5c57983a56c1ebd96ada0224b06ee31df8e1d916fbe2161cba97a6165f1"} Oct 30 06:11:01.970164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:01.969736 5980 scope.go:122] "RemoveContainer" containerID="22c0c7a8c47f8693bba21556cfd43582aa679b4a4022abe7f033d52f2c7cd610" Oct 30 06:11:02.034267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:02.034223 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz"] Oct 30 06:11:02.046766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:02.046725 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-85wzz"] Oct 30 06:11:02.589667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:02.589623 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:11:03.205949 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.205898 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg"] Oct 30 06:11:03.207036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.207003 5980 cpu_manager.go:414] "containerMap: removing container" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" Oct 30 06:11:03.207128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.207039 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" Oct 30 06:11:03.207128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.207075 5980 cpu_manager.go:414] "containerMap: removing container" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" Oct 30 06:11:03.207128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.207083 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" Oct 30 06:11:03.207268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.207253 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" Oct 30 06:11:03.207308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.207274 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="073b49be-33b8-4fa0-8d02-599933418225" containerName="istio-proxy" Oct 30 06:11:03.212982 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.212953 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.235861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.235816 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg"] Oct 30 06:11:03.240831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.240792 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-91f42c\"" Oct 30 06:11:03.241315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.241292 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-91f42c-dockercfg-fdz65\"" Oct 30 06:11:03.263284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.263217 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8"] Oct 30 06:11:03.269745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.269696 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.330733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.317987 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8"] Oct 30 06:11:03.369184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.349376 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lwc\" (UniqueName: \"kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc\") pod \"payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.392488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.392451 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.396134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.392793 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ccx\" (UniqueName: \"kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx\") pod \"payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.409426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.409388 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.547178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.540325 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lwc\" (UniqueName: \"kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc\") pod \"payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.547178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.540399 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.547178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.540457 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ccx\" (UniqueName: \"kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx\") pod \"payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.547178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.540525 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.547178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.545680 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.547178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.546856 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.575430 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.575395 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lwc\" (UniqueName: \"kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc\") pod \"payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:03.578270 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.578246 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ccx\" (UniqueName: \"kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx\") pod \"payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.633545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.633495 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:11:03.708011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.707969 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:03.848045 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:03.847964 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:04.019262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.018166 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073b49be-33b8-4fa0-8d02-599933418225" path="/var/lib/kubelet/pods/073b49be-33b8-4fa0-8d02-599933418225/volumes" Oct 30 06:11:04.099337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.099297 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8"] Oct 30 06:11:04.099521 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:11:04.099495 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3699cf25_4f57_42c8_b114_ff2fb561d189.slice/crio-922735f7db065bde0844885b3c80722d43ed89826c4ac2c4a0c21406c9c99cc8 WatchSource:0}: Error finding container 922735f7db065bde0844885b3c80722d43ed89826c4ac2c4a0c21406c9c99cc8: Status 404 returned error can't find the container with id 922735f7db065bde0844885b3c80722d43ed89826c4ac2c4a0c21406c9c99cc8 Oct 30 06:11:04.180737 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.180706 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg"] Oct 30 06:11:04.181494 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:11:04.181468 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa83e29_78fd_457d_911f_1a828ec86b4e.slice/crio-9654531910b5dd13005b74e19d3b02befb5897e2f5fff68bac18cd063c2d492f WatchSource:0}: Error finding container 9654531910b5dd13005b74e19d3b02befb5897e2f5fff68bac18cd063c2d492f: Status 404 returned error can't find the container with id 9654531910b5dd13005b74e19d3b02befb5897e2f5fff68bac18cd063c2d492f Oct 30 06:11:04.995924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.995881 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" event={"ID":"3699cf25-4f57-42c8-b114-ff2fb561d189","Type":"ContainerStarted","Data":"b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb"} Oct 30 06:11:04.996491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.995953 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:04.996491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.995978 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" event={"ID":"3699cf25-4f57-42c8-b114-ff2fb561d189","Type":"ContainerStarted","Data":"922735f7db065bde0844885b3c80722d43ed89826c4ac2c4a0c21406c9c99cc8"} Oct 30 06:11:04.997898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.997870 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" event={"ID":"dfa83e29-78fd-457d-911f-1a828ec86b4e","Type":"ContainerStarted","Data":"9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017"} Oct 30 06:11:04.998038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.997925 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:04.998038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:04.997942 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" event={"ID":"dfa83e29-78fd-457d-911f-1a828ec86b4e","Type":"ContainerStarted","Data":"9654531910b5dd13005b74e19d3b02befb5897e2f5fff68bac18cd063c2d492f"} Oct 30 06:11:05.043583 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:05.043530 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" podStartSLOduration=2.043512131 podStartE2EDuration="2.043512131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:05.040734716 +0000 UTC m=+2319.148773749" watchObservedRunningTime="2026-10-30 06:11:05.043512131 +0000 UTC m=+2319.151551166" Oct 30 06:11:05.085942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:05.085891 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" podStartSLOduration=2.085875941 podStartE2EDuration="2.085875941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:05.085032794 +0000 UTC m=+2319.193071827" watchObservedRunningTime="2026-10-30 06:11:05.085875941 +0000 UTC m=+2319.193914973" Oct 30 06:11:08.725173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.725139 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:11:08.790756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.786836 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf\" (UniqueName: \"kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf\") pod \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " Oct 30 06:11:08.790756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.786910 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume\") pod \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\" (UID: \"3674ab80-d2a1-4abe-8f8e-02dbe72b344a\") " Oct 30 06:11:08.794446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.794380 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume" pod "3674ab80-d2a1-4abe-8f8e-02dbe72b344a" (UID: "3674ab80-d2a1-4abe-8f8e-02dbe72b344a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:08.795101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.795077 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:11:08.800420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.800366 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf" pod "3674ab80-d2a1-4abe-8f8e-02dbe72b344a" (UID: "3674ab80-d2a1-4abe-8f8e-02dbe72b344a"). InnerVolumeSpecName "kube-api-access-c6snf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:08.927193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.926794 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk\" (UniqueName: \"kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk\") pod \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " Oct 30 06:11:08.927193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.926994 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume\") pod \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\" (UID: \"bdc2285c-ef59-4a3b-a138-d7018678fe3c\") " Oct 30 06:11:08.936650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.927961 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6snf\" (UniqueName: \"kubernetes.io/projected/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-kube-api-access-c6snf\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:08.936650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.927989 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3674ab80-d2a1-4abe-8f8e-02dbe72b344a-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:08.936650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.935523 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume" pod "bdc2285c-ef59-4a3b-a138-d7018678fe3c" (UID: "bdc2285c-ef59-4a3b-a138-d7018678fe3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:08.946600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:08.946550 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk" pod "bdc2285c-ef59-4a3b-a138-d7018678fe3c" (UID: "bdc2285c-ef59-4a3b-a138-d7018678fe3c"). InnerVolumeSpecName "kube-api-access-gsmbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:09.029051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029011 5980 generic.go:356] "Generic (PLEG): container finished" podID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerID="6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14" exitCode=0 Oct 30 06:11:09.029263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029074 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" Oct 30 06:11:09.029263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029090 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdc2285c-ef59-4a3b-a138-d7018678fe3c-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:09.029263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029130 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsmbk\" (UniqueName: \"kubernetes.io/projected/bdc2285c-ef59-4a3b-a138-d7018678fe3c-kube-api-access-gsmbk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:09.029263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029125 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" event={"ID":"3674ab80-d2a1-4abe-8f8e-02dbe72b344a","Type":"ContainerDied","Data":"6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14"} Oct 30 06:11:09.029263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029191 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb" event={"ID":"3674ab80-d2a1-4abe-8f8e-02dbe72b344a","Type":"ContainerDied","Data":"8c398073b209d035108c550bc543e21e44ca9dd79b71989f0c00c03dea04abad"} Oct 30 06:11:09.029263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.029227 5980 scope.go:122] "RemoveContainer" containerID="6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14" Oct 30 06:11:09.030756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.030732 5980 generic.go:356] "Generic (PLEG): container finished" podID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerID="a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e" exitCode=0 Oct 30 06:11:09.030887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.030785 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" Oct 30 06:11:09.030887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.030817 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" event={"ID":"bdc2285c-ef59-4a3b-a138-d7018678fe3c","Type":"ContainerDied","Data":"a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e"} Oct 30 06:11:09.030887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.030865 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln" event={"ID":"bdc2285c-ef59-4a3b-a138-d7018678fe3c","Type":"ContainerDied","Data":"0e6982c46b102264b6e242536c4e8fbf458d165ab2ac65d76f00837cdbe49218"} Oct 30 06:11:09.041473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.041439 5980 scope.go:122] "RemoveContainer" containerID="6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14" Oct 30 06:11:09.041796 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:11:09.041775 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14\": container with ID starting with 6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14 not found: ID does not exist" containerID="6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14" Oct 30 06:11:09.041863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.041806 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14"} err="failed to get container status \"6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14\": rpc error: code = NotFound desc = could not find container \"6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14\": container with ID starting with 6d6e78e1d59cf0ab400ece4750b1623b1bdb7b0f90573b7875c6737a3e03ab14 not found: ID does not exist" Oct 30 06:11:09.041863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.041825 5980 scope.go:122] "RemoveContainer" containerID="a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e" Oct 30 06:11:09.053766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.053743 5980 scope.go:122] "RemoveContainer" containerID="a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e" Oct 30 06:11:09.054149 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:11:09.054088 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e\": container with ID starting with a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e not found: ID does not exist" containerID="a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e" Oct 30 06:11:09.054234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.054139 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e"} err="failed to get container status \"a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e\": rpc error: code = NotFound desc = could not find container \"a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e\": container with ID starting with a1c515f8eda35b48ff60d5f73cd563a3e3d9754e2d8488933fac54a9a83ff43e not found: ID does not exist" Oct 30 06:11:09.085233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.085186 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln"] Oct 30 06:11:09.091649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.091612 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-62316c-544c69559-hcnln"] Oct 30 06:11:09.137589 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.137555 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb"] Oct 30 06:11:09.143231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:09.143200 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-62316c-866bb5d4b5-9mltb"] Oct 30 06:11:10.016736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:10.016693 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" path="/var/lib/kubelet/pods/3674ab80-d2a1-4abe-8f8e-02dbe72b344a/volumes" Oct 30 06:11:10.017285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:10.017268 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" path="/var/lib/kubelet/pods/bdc2285c-ef59-4a3b-a138-d7018678fe3c/volumes" Oct 30 06:11:13.093258 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:11:13.093218 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:11:15.218855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.218815 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq"] Oct 30 06:11:15.219509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.219492 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerName="payload-processing" Oct 30 06:11:15.219557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.219510 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerName="payload-processing" Oct 30 06:11:15.219593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.219564 5980 cpu_manager.go:414] "containerMap: removing container" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerName="payload-pre-processing" Oct 30 06:11:15.219593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.219569 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerName="payload-pre-processing" Oct 30 06:11:15.219665 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.219655 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3674ab80-d2a1-4abe-8f8e-02dbe72b344a" containerName="payload-processing" Oct 30 06:11:15.219696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.219677 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdc2285c-ef59-4a3b-a138-d7018678fe3c" containerName="payload-pre-processing" Oct 30 06:11:15.225336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.225308 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.259873 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.259821 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d0152-openshift-default-dockercfg-c9mvg\"" Oct 30 06:11:15.285138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.285077 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq"] Oct 30 06:11:15.388807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.388750 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.388807 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.388814 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389026 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389129 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389157 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389347 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389415 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389401 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389549 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.389617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.389603 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkdcl\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491021 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491074 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491210 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491244 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkdcl\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491395 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491447 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491478 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491568 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491586 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491623 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491654 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.491942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491814 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.492000 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.491942 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.492308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.492286 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.493892 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.493868 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.494027 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.494007 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.520380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.520342 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.520535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.520420 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkdcl\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.541955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.541923 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:15.887604 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.887563 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq"] Oct 30 06:11:15.887753 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:11:15.887658 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65ea9d1_565e_466d_b5a6_4cbc85196012.slice/crio-b8f917271c5f6f7cc3894bd51c5a18c4a303e8a6a09cf38aeebf749c2ee06410 WatchSource:0}: Error finding container b8f917271c5f6f7cc3894bd51c5a18c4a303e8a6a09cf38aeebf749c2ee06410: Status 404 returned error can't find the container with id b8f917271c5f6f7cc3894bd51c5a18c4a303e8a6a09cf38aeebf749c2ee06410 Oct 30 06:11:15.890652 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.890612 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:15.890784 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.890686 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:15.890784 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:15.890718 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:16.018622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.018592 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:11:16.018815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.018652 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:11:16.091373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.091335 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" event={"ID":"e65ea9d1-565e-466d-b5a6-4cbc85196012","Type":"ContainerStarted","Data":"ebbca4610a0bbc8761080848723b16779ffab4a2203b0a5319f199cd128a6e32"} Oct 30 06:11:16.091566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.091389 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" event={"ID":"e65ea9d1-565e-466d-b5a6-4cbc85196012","Type":"ContainerStarted","Data":"b8f917271c5f6f7cc3894bd51c5a18c4a303e8a6a09cf38aeebf749c2ee06410"} Oct 30 06:11:16.165788 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.165651 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podStartSLOduration=1.165625163 podStartE2EDuration="1.165625163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:16.159839534 +0000 UTC m=+2330.267878568" watchObservedRunningTime="2026-10-30 06:11:16.165625163 +0000 UTC m=+2330.273664199" Oct 30 06:11:16.542273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.542162 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:16.544492 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.544460 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:16.544685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:16.544519 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:17.543055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:17.543018 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:17.543616 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:17.543120 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:18.545595 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:18.545554 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:18.546225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:18.546189 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:19.002712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:19.002674 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5"] Oct 30 06:11:19.003255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:19.003214 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" containerID="cri-o://67185d812822c29b7fba9fb4f0e7db6315d882240c1afef0291db773aea47e03" gracePeriod=30 Oct 30 06:11:19.543288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:19.543245 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:19.543573 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:19.543317 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:20.544471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:20.544426 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:20.545015 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:20.544514 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:21.542934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:21.542894 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:21.543160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:21.542954 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:22.542898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:22.542862 5980 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 06:11:22.543349 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:22.542918 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 06:11:22.956736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:22.956693 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw"] Oct 30 06:11:22.957041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:22.956997 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" containerID="cri-o://08a50b0d4a629cf3597a740b4180343a2d44244cee53c7568f1f475f869da7c4" gracePeriod=30 Oct 30 06:11:23.547992 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:23.547961 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:23.548521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:23.548499 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:23.549440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:23.549418 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:11:24.324072 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.324036 5980 generic.go:356] "Generic (PLEG): container finished" podID="cdf3a979-6693-479c-9f19-0547f62be41b" containerID="67185d812822c29b7fba9fb4f0e7db6315d882240c1afef0291db773aea47e03" exitCode=0 Oct 30 06:11:24.324951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.324841 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" event={"ID":"cdf3a979-6693-479c-9f19-0547f62be41b","Type":"ContainerDied","Data":"67185d812822c29b7fba9fb4f0e7db6315d882240c1afef0291db773aea47e03"} Oct 30 06:11:24.393871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.393842 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:11:24.474793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.474758 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.475248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475224 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.475412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475400 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.475518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475503 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.475673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475659 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.475800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475784 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.475910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475897 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.476271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.475966 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:24.476271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.476082 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.476448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.476366 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket\") pod \"cdf3a979-6693-479c-9f19-0547f62be41b\" (UID: \"cdf3a979-6693-479c-9f19-0547f62be41b\") " Oct 30 06:11:24.481228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.479682 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.482020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.481979 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf3a979-6693-479c-9f19-0547f62be41b-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.482020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.482022 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.482310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.481985 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.482374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.482354 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.482483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.482387 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "kube-api-access-dlkrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:24.484779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.484732 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.484950 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.484838 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.485268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.485233 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:24.485440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.485415 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo" pod "cdf3a979-6693-479c-9f19-0547f62be41b" (UID: "cdf3a979-6693-479c-9f19-0547f62be41b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583862 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583904 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf3a979-6693-479c-9f19-0547f62be41b-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583915 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583925 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583936 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583945 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf3a979-6693-479c-9f19-0547f62be41b-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.583969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:24.583954 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkrd\" (UniqueName: \"kubernetes.io/projected/cdf3a979-6693-479c-9f19-0547f62be41b-kube-api-access-dlkrd\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:25.336656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:25.336626 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" Oct 30 06:11:25.336656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:25.336643 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5" event={"ID":"cdf3a979-6693-479c-9f19-0547f62be41b","Type":"ContainerDied","Data":"b3211d294539e1b30562e1db5b673d277c1b2b483e7d9253b0dd19041cb5f848"} Oct 30 06:11:25.336924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:25.336707 5980 scope.go:122] "RemoveContainer" containerID="67185d812822c29b7fba9fb4f0e7db6315d882240c1afef0291db773aea47e03" Oct 30 06:11:25.410044 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:25.410000 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5"] Oct 30 06:11:25.423398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:25.423360 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-djnq5"] Oct 30 06:11:26.048613 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:26.048577 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" path="/var/lib/kubelet/pods/cdf3a979-6693-479c-9f19-0547f62be41b/volumes" Oct 30 06:11:28.365932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.365896 5980 generic.go:356] "Generic (PLEG): container finished" podID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerID="08a50b0d4a629cf3597a740b4180343a2d44244cee53c7568f1f475f869da7c4" exitCode=0 Oct 30 06:11:28.365932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.365946 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" event={"ID":"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb","Type":"ContainerDied","Data":"08a50b0d4a629cf3597a740b4180343a2d44244cee53c7568f1f475f869da7c4"} Oct 30 06:11:28.718240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.718208 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:11:28.778311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.778265 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.779433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.778598 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.779433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.778700 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.779433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.778747 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.779433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.778778 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.779433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.779000 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.779433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.779318 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.779787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.779394 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.779787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.779523 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.780431 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.780334 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.780615 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.780539 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.781404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.780862 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:28.781404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.780918 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.781404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.780997 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo\") pod \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\" (UID: \"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb\") " Oct 30 06:11:28.782224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.782150 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.782224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.782175 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.782224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.782189 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.782224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.782203 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.782224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.782214 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.787618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.787583 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.788604 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.788569 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:28.789412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.789385 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "kube-api-access-2pvrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:28.789703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.789655 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token" pod "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" (UID: "3333e7fd-238e-4eeb-bcdb-0c948a73c8eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:28.883555 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.883520 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.883555 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.883552 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.883555 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.883561 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pvrv\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-kube-api-access-2pvrv\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.883800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:28.883570 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:11:29.374240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:29.374210 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" Oct 30 06:11:29.374721 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:29.374240 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw" event={"ID":"3333e7fd-238e-4eeb-bcdb-0c948a73c8eb","Type":"ContainerDied","Data":"2ab29e0714625c706f2168f367e8800058a1cd036342dcc3b77550bc3a59cf96"} Oct 30 06:11:29.374721 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:29.374319 5980 scope.go:122] "RemoveContainer" containerID="08a50b0d4a629cf3597a740b4180343a2d44244cee53c7568f1f475f869da7c4" Oct 30 06:11:29.446864 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:29.446825 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw"] Oct 30 06:11:29.459805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:29.459771 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-9gqmw"] Oct 30 06:11:30.016482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:11:30.016379 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" path="/var/lib/kubelet/pods/3333e7fd-238e-4eeb-bcdb-0c948a73c8eb/volumes" Oct 30 06:11:43.092059 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:11:43.091971 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:12:07.098663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.098608 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8"] Oct 30 06:12:07.100498 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.100473 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" Oct 30 06:12:07.100498 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.100500 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" Oct 30 06:12:07.100753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.100555 5980 cpu_manager.go:414] "containerMap: removing container" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" Oct 30 06:12:07.100753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.100564 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" Oct 30 06:12:07.101229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.100781 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3333e7fd-238e-4eeb-bcdb-0c948a73c8eb" containerName="istio-proxy" Oct 30 06:12:07.101229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.100805 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf3a979-6693-479c-9f19-0547f62be41b" containerName="istio-proxy" Oct 30 06:12:07.111239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.111217 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.128290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.128261 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8"] Oct 30 06:12:07.137061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.137033 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-db0540-openshift-default-dockercfg-2w8z8\"" Oct 30 06:12:07.209169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209075 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209209 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209268 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92sjq\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209502 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209536 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209569 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209689 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209602 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209689 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209646 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.209689 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.209671 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311582 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311660 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311692 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311725 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311768 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311790 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311882 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.311959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311934 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.312305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.311975 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92sjq\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.312498 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.312471 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.312637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.312498 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.312637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.312549 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.312637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.312628 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.312813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.312694 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.314448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.314420 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.314855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.314836 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.344662 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.343314 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.394176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.376951 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92sjq\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq\") pod \"e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.431134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.430387 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:07.894132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.894077 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8"] Oct 30 06:12:07.895338 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:12:07.895300 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc060397_36c4_4861_9c97_19eb24b3da3a.slice/crio-52ca5d112c998c67d9bb672e5958c9c4a278f2b76c038805f5fdee94cd928ea7 WatchSource:0}: Error finding container 52ca5d112c998c67d9bb672e5958c9c4a278f2b76c038805f5fdee94cd928ea7: Status 404 returned error can't find the container with id 52ca5d112c998c67d9bb672e5958c9c4a278f2b76c038805f5fdee94cd928ea7 Oct 30 06:12:07.899074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.898935 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:07.899074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.899059 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:07.899310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:07.899139 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:08.841134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:08.840809 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" event={"ID":"fc060397-36c4-4861-9c97-19eb24b3da3a","Type":"ContainerStarted","Data":"01839a1aafb985273da099aaded6e2a66344322117565f3b458ca34d280d9669"} Oct 30 06:12:08.841134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:08.840879 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" event={"ID":"fc060397-36c4-4861-9c97-19eb24b3da3a","Type":"ContainerStarted","Data":"52ca5d112c998c67d9bb672e5958c9c4a278f2b76c038805f5fdee94cd928ea7"} Oct 30 06:12:08.906525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:08.906468 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podStartSLOduration=1.9064537129999999 podStartE2EDuration="1.906453713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:08.906162017 +0000 UTC m=+2383.014201049" watchObservedRunningTime="2026-10-30 06:12:08.906453713 +0000 UTC m=+2383.014492745" Oct 30 06:12:09.431351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:09.431315 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:09.437635 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:09.437608 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:09.848315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:09.848283 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:09.849364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:09.849343 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:12:12.485015 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.484969 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8"] Oct 30 06:12:12.490225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.490192 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.523520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.523472 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8"] Oct 30 06:12:12.654195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654098 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654268 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654325 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654375 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654605 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654536 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654605 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654579 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxbk\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654638 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654729 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.654934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.654915 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.756924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.756821 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.756924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.756909 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.756952 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757082 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757139 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757340 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757363 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757422 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757338 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757454 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxbk\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757469 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757497 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.757550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.757539 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.758291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.758267 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.759857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.759835 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.760075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.760057 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.783446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.783412 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxbk\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.785660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.785634 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:12.808126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:12.808077 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:13.127857 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:12:13.127811 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:12:13.164215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.163946 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8"] Oct 30 06:12:13.164918 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:12:13.164853 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004a7fd2_38f9_4522_b994_73b7596e11c7.slice/crio-7316d04248a1c6dbc7908a1358588ad7e624d8c4043bd42e3aacc308651f3ab5 WatchSource:0}: Error finding container 7316d04248a1c6dbc7908a1358588ad7e624d8c4043bd42e3aacc308651f3ab5: Status 404 returned error can't find the container with id 7316d04248a1c6dbc7908a1358588ad7e624d8c4043bd42e3aacc308651f3ab5 Oct 30 06:12:13.169621 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.169569 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:13.169812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.169674 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:13.169812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.169718 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:13.298275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.298229 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk"] Oct 30 06:12:13.304514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.304475 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.323245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.322775 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-db0540-dockercfg-hz2qv\"" Oct 30 06:12:13.323245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.323125 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-db0540\"" Oct 30 06:12:13.329356 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.328563 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk"] Oct 30 06:12:13.597655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.597611 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.611345 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.611307 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4946d\" (UniqueName: \"kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d\") pod \"payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.730761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.730390 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.731185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.731101 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4946d\" (UniqueName: \"kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d\") pod \"payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.733217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.733187 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.760931 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.760888 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4946d\" (UniqueName: \"kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d\") pod \"payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.950553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.950461 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:13.969157 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.969066 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" event={"ID":"004a7fd2-38f9-4522-b994-73b7596e11c7","Type":"ContainerStarted","Data":"69943a8901e79c65754be8503c0b81ebc042b2c203bcfcf6703bd0a506a3bf78"} Oct 30 06:12:13.969331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:13.969164 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" event={"ID":"004a7fd2-38f9-4522-b994-73b7596e11c7","Type":"ContainerStarted","Data":"7316d04248a1c6dbc7908a1358588ad7e624d8c4043bd42e3aacc308651f3ab5"} Oct 30 06:12:14.086288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.086235 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podStartSLOduration=2.086217289 podStartE2EDuration="2.086217289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:14.083976621 +0000 UTC m=+2388.192015703" watchObservedRunningTime="2026-10-30 06:12:14.086217289 +0000 UTC m=+2388.194256379" Oct 30 06:12:14.328749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.328710 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk"] Oct 30 06:12:14.808539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.808448 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:14.810376 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.810346 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:12:14.810516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.810406 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 06:12:14.995199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.994999 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" event={"ID":"784231cb-7461-4b2e-9686-ab2c0fa47c8a","Type":"ContainerStarted","Data":"9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7"} Oct 30 06:12:14.995199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.995065 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" event={"ID":"784231cb-7461-4b2e-9686-ab2c0fa47c8a","Type":"ContainerStarted","Data":"71ade3fd88b1867ada919f3233107de349f44b46c0c87d86962213af6bf722ed"} Oct 30 06:12:14.995199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:14.995122 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:15.043799 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:15.043736 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" podStartSLOduration=2.04371651 podStartE2EDuration="2.04371651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:15.040457516 +0000 UTC m=+2389.148496550" watchObservedRunningTime="2026-10-30 06:12:15.04371651 +0000 UTC m=+2389.151755544" Oct 30 06:12:15.809666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:15.809631 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:12:15.810118 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:15.809682 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 06:12:16.808826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:16.808784 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:12:16.809004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:16.808857 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 06:12:17.809454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:17.809407 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:12:17.809948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:17.809480 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 06:12:18.809464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:18.809427 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:12:18.809950 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:18.809500 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 06:12:19.808913 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:19.808872 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 06:12:19.809166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:19.808944 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 06:12:20.814540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:20.814507 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:20.815035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:20.814967 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:20.815883 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:20.815862 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:12:24.402916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:24.402874 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8"] Oct 30 06:12:26.016581 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:26.016549 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:12:29.404222 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:29.404161 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerName="payload-processing" containerID="cri-o://c4127e08ad0c4ffc0721bd35f266547e0869fd2581577f098331694d2c23b374" gracePeriod=25 Oct 30 06:12:30.849984 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:30.849943 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:12:40.948593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:40.948550 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:12:43.122439 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:12:43.122400 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:12:45.408736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:45.408693 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf"] Oct 30 06:12:45.409705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:45.409666 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" containerID="cri-o://c7afd97fa2d3abac641ca69114f303a38772c77aac95d415eca8d319176c57a6" gracePeriod=30 Oct 30 06:12:46.232026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:46.231990 5980 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:12:46.232232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:46.232054 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:12:49.731946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.731863 5980 generic.go:356] "Generic (PLEG): container finished" podID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerID="c4127e08ad0c4ffc0721bd35f266547e0869fd2581577f098331694d2c23b374" exitCode=0 Oct 30 06:12:49.732370 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.731933 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" event={"ID":"d06b7249-21eb-435a-aa4d-3c6b5c6e4014","Type":"ContainerDied","Data":"c4127e08ad0c4ffc0721bd35f266547e0869fd2581577f098331694d2c23b374"} Oct 30 06:12:49.802384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.802343 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:12:49.919898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.919860 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume\") pod \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " Oct 30 06:12:49.920077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.920001 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh\" (UniqueName: \"kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh\") pod \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\" (UID: \"d06b7249-21eb-435a-aa4d-3c6b5c6e4014\") " Oct 30 06:12:49.920378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.920353 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume" pod "d06b7249-21eb-435a-aa4d-3c6b5c6e4014" (UID: "d06b7249-21eb-435a-aa4d-3c6b5c6e4014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:49.920736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.920635 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:49.922340 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:49.922318 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh" pod "d06b7249-21eb-435a-aa4d-3c6b5c6e4014" (UID: "d06b7249-21eb-435a-aa4d-3c6b5c6e4014"). InnerVolumeSpecName "kube-api-access-8bdkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:50.031716 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.031578 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdkh\" (UniqueName: \"kubernetes.io/projected/d06b7249-21eb-435a-aa4d-3c6b5c6e4014-kube-api-access-8bdkh\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.744509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.744476 5980 generic.go:356] "Generic (PLEG): container finished" podID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerID="c7afd97fa2d3abac641ca69114f303a38772c77aac95d415eca8d319176c57a6" exitCode=0 Oct 30 06:12:50.744946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.744570 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" event={"ID":"0a388c82-7c52-47fc-bda2-f77c7a3c60c2","Type":"ContainerDied","Data":"c7afd97fa2d3abac641ca69114f303a38772c77aac95d415eca8d319176c57a6"} Oct 30 06:12:50.744946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.744634 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" event={"ID":"0a388c82-7c52-47fc-bda2-f77c7a3c60c2","Type":"ContainerDied","Data":"1db78c4d51c3c17257ee3201d354e42474c826a65f7dbe5b52f081077f395109"} Oct 30 06:12:50.744946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.744658 5980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db78c4d51c3c17257ee3201d354e42474c826a65f7dbe5b52f081077f395109" Oct 30 06:12:50.746368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.746338 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" Oct 30 06:12:50.746518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.746369 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8" event={"ID":"d06b7249-21eb-435a-aa4d-3c6b5c6e4014","Type":"ContainerDied","Data":"70ad9697be3936e1c658d270c92eafc097248d31bdc22c89c43c76ef011b23b7"} Oct 30 06:12:50.746518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.746422 5980 scope.go:122] "RemoveContainer" containerID="c4127e08ad0c4ffc0721bd35f266547e0869fd2581577f098331694d2c23b374" Oct 30 06:12:50.790313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.789740 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:12:50.861235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861196 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861413 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861471 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861579 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861716 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861700 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861786 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861771 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861803 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861834 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861903 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs\") pod \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\" (UID: \"0a388c82-7c52-47fc-bda2-f77c7a3c60c2\") " Oct 30 06:12:50.861945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.861913 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:50.862733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.862706 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.863237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.863211 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.863378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.863297 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.863514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.863488 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.866158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.866074 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:12:50.866450 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.866425 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "kube-api-access-dllwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:50.867227 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.867201 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.903375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.903327 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8"] Oct 30 06:12:50.919718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.919672 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d3e7e6-5d745d757c-xt4c8"] Oct 30 06:12:50.926368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.926332 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964444 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964476 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964537 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllwd\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-kube-api-access-dllwd\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964551 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964589 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964602 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.979312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:50.964614 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.016142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.012228 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token" pod "0a388c82-7c52-47fc-bda2-f77c7a3c60c2" (UID: "0a388c82-7c52-47fc-bda2-f77c7a3c60c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:51.065878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.065839 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a388c82-7c52-47fc-bda2-f77c7a3c60c2-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.755789 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.755698 5980 generic.go:356] "Generic (PLEG): container finished" podID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerID="c528fbd23330b3b9f3168ea8220390c3daf0b57c8d26c9721c6a24738ff7adcf" exitCode=0 Oct 30 06:12:51.755789 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.755767 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" event={"ID":"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8","Type":"ContainerDied","Data":"c528fbd23330b3b9f3168ea8220390c3daf0b57c8d26c9721c6a24738ff7adcf"} Oct 30 06:12:51.756263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.755900 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf" Oct 30 06:12:51.756440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.756425 5980 scope.go:122] "RemoveContainer" containerID="c528fbd23330b3b9f3168ea8220390c3daf0b57c8d26c9721c6a24738ff7adcf" Oct 30 06:12:51.758191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.758160 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:51.758727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.758229 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:51.758727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.758263 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:51.829652 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.829613 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf"] Oct 30 06:12:51.841624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:51.841595 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-lbddf"] Oct 30 06:12:52.021361 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:52.021286 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" path="/var/lib/kubelet/pods/0a388c82-7c52-47fc-bda2-f77c7a3c60c2/volumes" Oct 30 06:12:52.021806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:52.021791 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" path="/var/lib/kubelet/pods/d06b7249-21eb-435a-aa4d-3c6b5c6e4014/volumes" Oct 30 06:12:52.788133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:52.781897 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" event={"ID":"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8","Type":"ContainerStarted","Data":"6e55585e707a0c780d6a9b945c0e72dd216ff928e8db3b957da5a5d612ded8eb"} Oct 30 06:12:53.423761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:53.423716 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:12:53.424032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:53.424008 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 06:12:53.424080 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:53.424060 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 06:12:54.424924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:54.424880 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 06:12:54.425400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:54.424951 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 06:12:55.424869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:55.424837 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 06:12:55.425067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:55.424890 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 06:12:56.295919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:56.295882 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr"] Oct 30 06:12:56.319273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:56.319233 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc"] Oct 30 06:12:56.429062 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:56.428608 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 06:12:56.429062 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:56.428669 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 06:12:57.424642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:57.424608 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 06:12:57.424846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:57.424667 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 06:12:58.050609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.050572 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29"] Oct 30 06:12:58.051309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.051292 5980 cpu_manager.go:414] "containerMap: removing container" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" Oct 30 06:12:58.051309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.051310 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" Oct 30 06:12:58.051394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.051319 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerName="payload-processing" Oct 30 06:12:58.051394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.051324 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerName="payload-processing" Oct 30 06:12:58.051470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.051459 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06b7249-21eb-435a-aa4d-3c6b5c6e4014" containerName="payload-processing" Oct 30 06:12:58.051506 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.051484 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a388c82-7c52-47fc-bda2-f77c7a3c60c2" containerName="istio-proxy" Oct 30 06:12:58.054586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.054562 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.080814 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.080778 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29"] Oct 30 06:12:58.095087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.095048 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-477f6b-openshift-default-dockercfg-q6lhv\"" Oct 30 06:12:58.154522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154490 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.154751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154629 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.154751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154689 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.154751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154723 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.154946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154796 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6r8v\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.154946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154872 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.154946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154915 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.155186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.154982 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.155186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.155016 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256120 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256214 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256260 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256499 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256407 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256499 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256446 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256499 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256539 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256588 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256611 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6r8v\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256630 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256712 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.256777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.256760 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.257164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.257139 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.257656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.257629 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.259089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.259015 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.259619 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.259598 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.282754 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.282725 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6r8v\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.283226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.283200 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.372386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.372342 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:58.430517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.430486 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:12:58.430904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.430881 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:12:58.432945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.432920 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:12:58.717368 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:12:58.717339 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478a125a_bb6b_4338_9d02_5904cb583511.slice/crio-bc7d20b82656dcb4e01373f0e45f691279ed703d75ca8991316c6e646acb101e WatchSource:0}: Error finding container bc7d20b82656dcb4e01373f0e45f691279ed703d75ca8991316c6e646acb101e: Status 404 returned error can't find the container with id bc7d20b82656dcb4e01373f0e45f691279ed703d75ca8991316c6e646acb101e Oct 30 06:12:58.717536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.717513 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29"] Oct 30 06:12:58.720237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.720204 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:58.720324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.720278 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:58.720324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.720308 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:58.917156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.916433 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" event={"ID":"478a125a-bb6b-4338-9d02-5904cb583511","Type":"ContainerStarted","Data":"3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12"} Oct 30 06:12:58.917156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.916504 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" event={"ID":"478a125a-bb6b-4338-9d02-5904cb583511","Type":"ContainerStarted","Data":"bc7d20b82656dcb4e01373f0e45f691279ed703d75ca8991316c6e646acb101e"} Oct 30 06:12:58.967251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:58.967123 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podStartSLOduration=0.967083598 podStartE2EDuration="967.083598ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:58.965897283 +0000 UTC m=+2433.073936317" watchObservedRunningTime="2026-10-30 06:12:58.967083598 +0000 UTC m=+2433.075122632" Oct 30 06:12:59.372667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:59.372629 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:12:59.374814 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:59.374783 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:12:59.374966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:12:59.374836 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:00.382134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:00.374565 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:13:00.382134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:00.374628 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:01.296777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.296716 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" podUID="ff32e94e-a308-4306-b0f0-75c67f6832d7" containerName="payload-processing" containerID="cri-o://30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771" gracePeriod=25 Oct 30 06:13:01.319879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.319842 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerName="payload-pre-processing" containerID="cri-o://11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c" gracePeriod=25 Oct 30 06:13:01.387448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.387407 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:13:01.388857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.388825 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:01.649892 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.649861 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:13:01.749198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.749155 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume\") pod \"ff32e94e-a308-4306-b0f0-75c67f6832d7\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " Oct 30 06:13:01.749410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.749345 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7\" (UniqueName: \"kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7\") pod \"ff32e94e-a308-4306-b0f0-75c67f6832d7\" (UID: \"ff32e94e-a308-4306-b0f0-75c67f6832d7\") " Oct 30 06:13:01.749628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.749604 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume" pod "ff32e94e-a308-4306-b0f0-75c67f6832d7" (UID: "ff32e94e-a308-4306-b0f0-75c67f6832d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:01.749964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.749943 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff32e94e-a308-4306-b0f0-75c67f6832d7-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:01.751808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.751783 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7" pod "ff32e94e-a308-4306-b0f0-75c67f6832d7" (UID: "ff32e94e-a308-4306-b0f0-75c67f6832d7"). InnerVolumeSpecName "kube-api-access-qzwk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:01.851069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.851039 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwk7\" (UniqueName: \"kubernetes.io/projected/ff32e94e-a308-4306-b0f0-75c67f6832d7-kube-api-access-qzwk7\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:01.947406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.947308 5980 generic.go:356] "Generic (PLEG): container finished" podID="ff32e94e-a308-4306-b0f0-75c67f6832d7" containerID="30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771" exitCode=0 Oct 30 06:13:01.947584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.947424 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" event={"ID":"ff32e94e-a308-4306-b0f0-75c67f6832d7","Type":"ContainerDied","Data":"30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771"} Oct 30 06:13:01.947584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.947514 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" event={"ID":"ff32e94e-a308-4306-b0f0-75c67f6832d7","Type":"ContainerDied","Data":"c78611928d77df678e8a5786c8809027962eea86286f88452c4bdd4d0138c811"} Oct 30 06:13:01.947584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.947452 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr" Oct 30 06:13:01.947765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.947546 5980 scope.go:122] "RemoveContainer" containerID="30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771" Oct 30 06:13:01.995928 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.995907 5980 scope.go:122] "RemoveContainer" containerID="30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771" Oct 30 06:13:01.996369 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:13:01.996343 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771\": container with ID starting with 30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771 not found: ID does not exist" containerID="30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771" Oct 30 06:13:01.996453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:01.996380 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771"} err="failed to get container status \"30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771\": rpc error: code = NotFound desc = could not find container \"30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771\": container with ID starting with 30e327f0281317d4015548f10ddc1a451213019f7f906c3ce51b072ed9f1d771 not found: ID does not exist" Oct 30 06:13:02.035709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:02.035681 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr"] Oct 30 06:13:02.039885 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:02.039852 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d0152-9fb5f5555-8pstr"] Oct 30 06:13:02.384134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:02.383208 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:13:02.384134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:02.383267 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:03.384134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:03.383556 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:13:03.384134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:03.383621 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:04.017799 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:04.017764 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff32e94e-a308-4306-b0f0-75c67f6832d7" path="/var/lib/kubelet/pods/ff32e94e-a308-4306-b0f0-75c67f6832d7/volumes" Oct 30 06:13:04.374626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:04.374582 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:13:04.374940 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:04.374910 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:05.376557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:05.376518 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 06:13:05.377325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:05.377284 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 06:13:06.378692 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:06.378650 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:13:06.379094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:06.379065 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:13:06.379804 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:06.379786 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:13:08.287407 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:08.287369 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:13:13.091077 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:13:13.091037 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:13:18.286866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:18.286831 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:13:21.619480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.619454 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:13:21.718691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.718651 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume\") pod \"87c625ed-ad61-4c28-9e97-db1fde32c265\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " Oct 30 06:13:21.718882 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.718733 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf\" (UniqueName: \"kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf\") pod \"87c625ed-ad61-4c28-9e97-db1fde32c265\" (UID: \"87c625ed-ad61-4c28-9e97-db1fde32c265\") " Oct 30 06:13:21.719090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.719066 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume" pod "87c625ed-ad61-4c28-9e97-db1fde32c265" (UID: "87c625ed-ad61-4c28-9e97-db1fde32c265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:21.719700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.719676 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87c625ed-ad61-4c28-9e97-db1fde32c265-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:21.721446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.721414 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf" pod "87c625ed-ad61-4c28-9e97-db1fde32c265" (UID: "87c625ed-ad61-4c28-9e97-db1fde32c265"). InnerVolumeSpecName "kube-api-access-sljwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:21.821288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:21.821263 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljwf\" (UniqueName: \"kubernetes.io/projected/87c625ed-ad61-4c28-9e97-db1fde32c265-kube-api-access-sljwf\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:22.107502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.107467 5980 generic.go:356] "Generic (PLEG): container finished" podID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerID="11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c" exitCode=0 Oct 30 06:13:22.107678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.107522 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" Oct 30 06:13:22.107678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.107549 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" event={"ID":"87c625ed-ad61-4c28-9e97-db1fde32c265","Type":"ContainerDied","Data":"11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c"} Oct 30 06:13:22.107678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.107610 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc" event={"ID":"87c625ed-ad61-4c28-9e97-db1fde32c265","Type":"ContainerDied","Data":"8ebed4bfc9450c20d61ec2154b8abaf20599b1e61a46226e7818c0303123150f"} Oct 30 06:13:22.107678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.107634 5980 scope.go:122] "RemoveContainer" containerID="11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c" Oct 30 06:13:22.118690 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.118657 5980 scope.go:122] "RemoveContainer" containerID="11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c" Oct 30 06:13:22.118970 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:13:22.118947 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c\": container with ID starting with 11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c not found: ID does not exist" containerID="11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c" Oct 30 06:13:22.119050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.118977 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c"} err="failed to get container status \"11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c\": rpc error: code = NotFound desc = could not find container \"11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c\": container with ID starting with 11e882f4dcb7805c73c9c495beb17cdd56a9b64eeeefba4731c5afcecbaae45c not found: ID does not exist" Oct 30 06:13:22.167439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.167406 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc"] Oct 30 06:13:22.174951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:22.174919 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d0152-6c48d66bc9-5qtzc"] Oct 30 06:13:24.017527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:24.017495 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" path="/var/lib/kubelet/pods/87c625ed-ad61-4c28-9e97-db1fde32c265/volumes" Oct 30 06:13:28.203386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:28.203355 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk"] Oct 30 06:13:30.087210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:30.086497 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq"] Oct 30 06:13:30.087210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:30.086816 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" containerID="cri-o://ebbca4610a0bbc8761080848723b16779ffab4a2203b0a5319f199cd128a6e32" gracePeriod=30 Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.308643 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp"] Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.309383 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ff32e94e-a308-4306-b0f0-75c67f6832d7" containerName="payload-processing" Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.309398 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff32e94e-a308-4306-b0f0-75c67f6832d7" containerName="payload-processing" Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.309443 5980 cpu_manager.go:414] "containerMap: removing container" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerName="payload-pre-processing" Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.309448 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerName="payload-pre-processing" Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.309547 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff32e94e-a308-4306-b0f0-75c67f6832d7" containerName="payload-processing" Oct 30 06:13:31.309579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.309563 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c625ed-ad61-4c28-9e97-db1fde32c265" containerName="payload-pre-processing" Oct 30 06:13:31.314846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.314824 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.342578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.342539 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.342841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.342816 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.342984 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.342969 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqx9q\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.343142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.343127 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.343680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.343657 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.353277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.352178 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.353277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.352288 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.353277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.352325 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.353277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.352357 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.353277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.346324 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-25d881-openshift-default-dockercfg-8djl9\"" Oct 30 06:13:31.353277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.352014 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp"] Oct 30 06:13:31.453890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.453855 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.453930 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454014 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454044 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454064 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454102 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454147 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454187 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqx9q\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454318 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454359 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454828 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454780 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.454933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454843 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.455008 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454979 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.455008 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.454996 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.456864 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.456844 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.457227 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.457206 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.505766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.505728 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.543132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.535837 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqx9q\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:31.649132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:31.648261 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:32.201204 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:32.200934 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp"] Oct 30 06:13:32.201348 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:13:32.201183 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a254368_d09a_4e4a_a6e3_2f2e1446c64a.slice/crio-0babcfcd8e8875cb760fe3ecde547fb3308fbc8f787e17b99c3146c819200385 WatchSource:0}: Error finding container 0babcfcd8e8875cb760fe3ecde547fb3308fbc8f787e17b99c3146c819200385: Status 404 returned error can't find the container with id 0babcfcd8e8875cb760fe3ecde547fb3308fbc8f787e17b99c3146c819200385 Oct 30 06:13:32.204177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:32.204142 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:32.204259 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:32.204209 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:32.204259 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:32.204240 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:32.231769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:32.231738 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" event={"ID":"1a254368-d09a-4e4a-a6e3-2f2e1446c64a","Type":"ContainerStarted","Data":"0babcfcd8e8875cb760fe3ecde547fb3308fbc8f787e17b99c3146c819200385"} Oct 30 06:13:33.204525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.204474 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" podUID="784231cb-7461-4b2e-9686-ab2c0fa47c8a" containerName="payload-pre-processing" containerID="cri-o://9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7" gracePeriod=25 Oct 30 06:13:33.239497 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.239465 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" event={"ID":"1a254368-d09a-4e4a-a6e3-2f2e1446c64a","Type":"ContainerStarted","Data":"2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4"} Oct 30 06:13:33.315145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.308094 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podStartSLOduration=2.308075471 podStartE2EDuration="2.308075471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:33.307405133 +0000 UTC m=+2467.415444168" watchObservedRunningTime="2026-10-30 06:13:33.308075471 +0000 UTC m=+2467.416114505" Oct 30 06:13:33.649542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.649508 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:33.677136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.661386 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:33.745149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.745077 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:13:33.844040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.844013 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d\" (UniqueName: \"kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d\") pod \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " Oct 30 06:13:33.844275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.844256 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume\") pod \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\" (UID: \"784231cb-7461-4b2e-9686-ab2c0fa47c8a\") " Oct 30 06:13:33.844994 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.844633 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume" pod "784231cb-7461-4b2e-9686-ab2c0fa47c8a" (UID: "784231cb-7461-4b2e-9686-ab2c0fa47c8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:33.844994 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.844970 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784231cb-7461-4b2e-9686-ab2c0fa47c8a-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:33.846476 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.846455 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d" pod "784231cb-7461-4b2e-9686-ab2c0fa47c8a" (UID: "784231cb-7461-4b2e-9686-ab2c0fa47c8a"). InnerVolumeSpecName "kube-api-access-4946d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:33.946414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:33.946337 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4946d\" (UniqueName: \"kubernetes.io/projected/784231cb-7461-4b2e-9686-ab2c0fa47c8a-kube-api-access-4946d\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:34.248753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.248675 5980 generic.go:356] "Generic (PLEG): container finished" podID="784231cb-7461-4b2e-9686-ab2c0fa47c8a" containerID="9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7" exitCode=0 Oct 30 06:13:34.248753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.248730 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" event={"ID":"784231cb-7461-4b2e-9686-ab2c0fa47c8a","Type":"ContainerDied","Data":"9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7"} Oct 30 06:13:34.249261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.248737 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" Oct 30 06:13:34.249261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.248807 5980 scope.go:122] "RemoveContainer" containerID="9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7" Oct 30 06:13:34.249261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.248794 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk" event={"ID":"784231cb-7461-4b2e-9686-ab2c0fa47c8a","Type":"ContainerDied","Data":"71ade3fd88b1867ada919f3233107de349f44b46c0c87d86962213af6bf722ed"} Oct 30 06:13:34.249261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.249244 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:34.250417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.250272 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:13:34.260079 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.260063 5980 scope.go:122] "RemoveContainer" containerID="9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7" Oct 30 06:13:34.260400 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:13:34.260382 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7\": container with ID starting with 9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7 not found: ID does not exist" containerID="9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7" Oct 30 06:13:34.260458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.260409 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7"} err="failed to get container status \"9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7\": rpc error: code = NotFound desc = could not find container \"9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7\": container with ID starting with 9afc4ffceaefb1fff2017cd58bcd6fca95c7b7d49cb0e416f7c8c1f724e9dcc7 not found: ID does not exist" Oct 30 06:13:34.404154 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.401231 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk"] Oct 30 06:13:34.406275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:34.406247 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-db0540-77c6d9955f-rzdxk"] Oct 30 06:13:35.258329 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.258293 5980 generic.go:356] "Generic (PLEG): container finished" podID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerID="ebbca4610a0bbc8761080848723b16779ffab4a2203b0a5319f199cd128a6e32" exitCode=0 Oct 30 06:13:35.258697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.258465 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" event={"ID":"e65ea9d1-565e-466d-b5a6-4cbc85196012","Type":"ContainerDied","Data":"ebbca4610a0bbc8761080848723b16779ffab4a2203b0a5319f199cd128a6e32"} Oct 30 06:13:35.487186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.475265 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:13:35.621377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.621339 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.622490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.622466 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.622728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.621904 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.622728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.622700 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.622952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.622742 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.622952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.622774 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.622952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.622888 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.623149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.623034 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.623149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.623067 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.623284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.623189 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert\") pod \"e65ea9d1-565e-466d-b5a6-4cbc85196012\" (UID: \"e65ea9d1-565e-466d-b5a6-4cbc85196012\") " Oct 30 06:13:35.624346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.623569 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.624346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.623961 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.624346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.623983 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.624569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.624344 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.624898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.624874 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.625087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.625054 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:35.630674 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.630652 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:35.645890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.635222 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "kube-api-access-rkdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:35.645890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.635289 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:13:35.645890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.635340 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy" pod "e65ea9d1-565e-466d-b5a6-4cbc85196012" (UID: "e65ea9d1-565e-466d-b5a6-4cbc85196012"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.725848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725810 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.725848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725847 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.726478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725865 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65ea9d1-565e-466d-b5a6-4cbc85196012-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.726478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725878 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.726478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725891 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65ea9d1-565e-466d-b5a6-4cbc85196012-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.726478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725904 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkdcl\" (UniqueName: \"kubernetes.io/projected/e65ea9d1-565e-466d-b5a6-4cbc85196012-kube-api-access-rkdcl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.726478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:35.725918 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65ea9d1-565e-466d-b5a6-4cbc85196012-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:36.017603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:36.017528 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784231cb-7461-4b2e-9686-ab2c0fa47c8a" path="/var/lib/kubelet/pods/784231cb-7461-4b2e-9686-ab2c0fa47c8a/volumes" Oct 30 06:13:36.449961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:36.449926 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" Oct 30 06:13:36.450756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:36.450725 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq" event={"ID":"e65ea9d1-565e-466d-b5a6-4cbc85196012","Type":"ContainerDied","Data":"b8f917271c5f6f7cc3894bd51c5a18c4a303e8a6a09cf38aeebf749c2ee06410"} Oct 30 06:13:36.450985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:36.450968 5980 scope.go:122] "RemoveContainer" containerID="ebbca4610a0bbc8761080848723b16779ffab4a2203b0a5319f199cd128a6e32" Oct 30 06:13:36.554931 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:36.554837 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq"] Oct 30 06:13:36.593630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:36.593532 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-9brmq"] Oct 30 06:13:37.499813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.499777 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v"] Oct 30 06:13:37.500867 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.500842 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" Oct 30 06:13:37.500974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.500868 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" Oct 30 06:13:37.500974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.500914 5980 cpu_manager.go:414] "containerMap: removing container" podUID="784231cb-7461-4b2e-9686-ab2c0fa47c8a" containerName="payload-pre-processing" Oct 30 06:13:37.500974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.500924 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784231cb-7461-4b2e-9686-ab2c0fa47c8a" containerName="payload-pre-processing" Oct 30 06:13:37.501190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.501147 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" containerName="istio-proxy" Oct 30 06:13:37.501190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.501167 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="784231cb-7461-4b2e-9686-ab2c0fa47c8a" containerName="payload-pre-processing" Oct 30 06:13:37.507445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.507422 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:37.528845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.528823 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v"] Oct 30 06:13:37.536668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.536639 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-25d881-dockercfg-bvlcj\"" Oct 30 06:13:37.561921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.561883 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7fdb\" (UniqueName: \"kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb\") pod \"payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:37.562250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.562219 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:37.606603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.606569 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-25d881\"" Oct 30 06:13:37.693134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.679546 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7fdb\" (UniqueName: \"kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb\") pod \"payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:37.693134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.679721 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:37.693134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.680998 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:37.865158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:37.863779 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7fdb\" (UniqueName: \"kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb\") pod \"payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:38.016977 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:38.016944 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65ea9d1-565e-466d-b5a6-4cbc85196012" path="/var/lib/kubelet/pods/e65ea9d1-565e-466d-b5a6-4cbc85196012/volumes" Oct 30 06:13:38.132235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:38.132152 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:38.449617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:38.449587 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v"] Oct 30 06:13:38.450510 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:13:38.450471 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96ac33d_eca1_4daf_8e6f_f71c41c216d0.slice/crio-0f5d71541c13833c3ce0653cc4ac09504b67db600e304ccbfff93b26ca80fe90 WatchSource:0}: Error finding container 0f5d71541c13833c3ce0653cc4ac09504b67db600e304ccbfff93b26ca80fe90: Status 404 returned error can't find the container with id 0f5d71541c13833c3ce0653cc4ac09504b67db600e304ccbfff93b26ca80fe90 Oct 30 06:13:38.469804 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:38.469772 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" event={"ID":"f96ac33d-eca1-4daf-8e6f-f71c41c216d0","Type":"ContainerStarted","Data":"0f5d71541c13833c3ce0653cc4ac09504b67db600e304ccbfff93b26ca80fe90"} Oct 30 06:13:39.482766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:39.482728 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" event={"ID":"f96ac33d-eca1-4daf-8e6f-f71c41c216d0","Type":"ContainerStarted","Data":"41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642"} Oct 30 06:13:39.483304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:39.482930 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:39.531465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:39.531398 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" podStartSLOduration=2.531377507 podStartE2EDuration="2.531377507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:39.526027753 +0000 UTC m=+2473.634066814" watchObservedRunningTime="2026-10-30 06:13:39.531377507 +0000 UTC m=+2473.639416544" Oct 30 06:13:40.890249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:40.890213 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:13:40.890871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:40.890275 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:13:42.300332 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.300298 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr"] Oct 30 06:13:42.304732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.304705 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.325366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.325339 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dddc53\"" Oct 30 06:13:42.330724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.330696 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr"] Oct 30 06:13:42.369600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.369569 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.369823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.369805 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgbb\" (UniqueName: \"kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb\") pod \"payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.378557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.378532 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k"] Oct 30 06:13:42.381711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.381695 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:42.414999 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.414973 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dddc53-dockercfg-4d7fc\"" Oct 30 06:13:42.417418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.417396 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k"] Oct 30 06:13:42.488148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.486937 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:42.488148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.487033 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgbb\" (UniqueName: \"kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb\") pod \"payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.488148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.487121 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpsvg\" (UniqueName: \"kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:42.488148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.487294 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.488529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.488303 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.639133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.621247 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgbb\" (UniqueName: \"kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb\") pod \"payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.648132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.647826 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:42.648132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.647941 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpsvg\" (UniqueName: \"kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:42.663129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.654617 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:42.671173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.671139 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:42.693585 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.693549 5980 generic.go:356] "Generic (PLEG): container finished" podID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerID="01839a1aafb985273da099aaded6e2a66344322117565f3b458ca34d280d9669" exitCode=0 Oct 30 06:13:42.693675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.693650 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" event={"ID":"fc060397-36c4-4861-9c97-19eb24b3da3a","Type":"ContainerDied","Data":"01839a1aafb985273da099aaded6e2a66344322117565f3b458ca34d280d9669"} Oct 30 06:13:42.694156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.694135 5980 scope.go:122] "RemoveContainer" containerID="01839a1aafb985273da099aaded6e2a66344322117565f3b458ca34d280d9669" Oct 30 06:13:42.696423 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.696326 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:42.698420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.696423 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:42.698420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.696469 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:42.721473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.721446 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpsvg\" (UniqueName: \"kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:43.002150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:42.998128 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:43.002150 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:13:42.999752 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723630fe_6645_486c_8e7c_264fa375ab6d.slice/crio-209c7b120400f69134ffa5a2c6ccf91a5013aa271743a1e32c370c802911b1b2 WatchSource:0}: Error finding container 209c7b120400f69134ffa5a2c6ccf91a5013aa271743a1e32c370c802911b1b2: Status 404 returned error can't find the container with id 209c7b120400f69134ffa5a2c6ccf91a5013aa271743a1e32c370c802911b1b2 Oct 30 06:13:43.002566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.002392 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr"] Oct 30 06:13:43.146678 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:13:43.144397 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:13:43.377608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.377576 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k"] Oct 30 06:13:43.405092 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:13:43.405058 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d877394_f587_4642_a895_75140b325d02.slice/crio-4b9d3f3fea94df76077bf4fcdb7c0676f4f6db5fab581d29af7dc8170377eb70 WatchSource:0}: Error finding container 4b9d3f3fea94df76077bf4fcdb7c0676f4f6db5fab581d29af7dc8170377eb70: Status 404 returned error can't find the container with id 4b9d3f3fea94df76077bf4fcdb7c0676f4f6db5fab581d29af7dc8170377eb70 Oct 30 06:13:43.703763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.703665 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" event={"ID":"723630fe-6645-486c-8e7c-264fa375ab6d","Type":"ContainerStarted","Data":"640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a"} Oct 30 06:13:43.703763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.703738 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" event={"ID":"723630fe-6645-486c-8e7c-264fa375ab6d","Type":"ContainerStarted","Data":"209c7b120400f69134ffa5a2c6ccf91a5013aa271743a1e32c370c802911b1b2"} Oct 30 06:13:43.704598 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.704574 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:43.706262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.706231 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" event={"ID":"fc060397-36c4-4861-9c97-19eb24b3da3a","Type":"ContainerStarted","Data":"07afd0ac45644d744e82d61bd210edfe849452bcaa09095d7361f0b5c4dd30c1"} Oct 30 06:13:43.708094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.708065 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" event={"ID":"8d877394-f587-4642-a895-75140b325d02","Type":"ContainerStarted","Data":"cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00"} Oct 30 06:13:43.708255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.708139 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" event={"ID":"8d877394-f587-4642-a895-75140b325d02","Type":"ContainerStarted","Data":"4b9d3f3fea94df76077bf4fcdb7c0676f4f6db5fab581d29af7dc8170377eb70"} Oct 30 06:13:43.708255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.708181 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:43.783239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.783177 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" podStartSLOduration=1.783158172 podStartE2EDuration="1.783158172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:43.777055788 +0000 UTC m=+2477.885094824" watchObservedRunningTime="2026-10-30 06:13:43.783158172 +0000 UTC m=+2477.891197223" Oct 30 06:13:43.891136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:43.879202 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" podStartSLOduration=1.8791819140000001 podStartE2EDuration="1.879181914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:43.877552544 +0000 UTC m=+2477.985591585" watchObservedRunningTime="2026-10-30 06:13:43.879181914 +0000 UTC m=+2477.987220950" Oct 30 06:13:44.431464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:44.431435 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:13:44.431891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:44.431722 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:13:44.431891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:44.431776 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 06:13:45.431234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:45.431202 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:13:45.431421 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:45.431258 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 06:13:46.432029 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.431995 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:13:46.432490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.432054 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 06:13:46.493136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.493005 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7"] Oct 30 06:13:46.503067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.502882 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.513699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.513665 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7"] Oct 30 06:13:46.524563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.523738 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-d1bfce-openshift-default-dockercfg-xxfmp\"" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556014 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556146 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556196 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556312 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556380 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556504 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556585 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg4qh\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556734 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.557173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.556761 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658090 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658173 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658225 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658285 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658446 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658501 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658559 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg4qh\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658693 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658714 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658748 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.658876 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.658802 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.659250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.659230 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.659384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.659239 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.659467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.659284 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.661390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.661367 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.661724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.661708 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.688168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.687605 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.688168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.687955 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg4qh\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:46.829572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:46.829535 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:47.210124 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.209189 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7"] Oct 30 06:13:47.214249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.214207 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:47.214375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.214304 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:47.214375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.214351 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:47.431118 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.431070 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:13:47.431286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.431148 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 06:13:47.733705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.733383 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk"] Oct 30 06:13:47.733705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.733645 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" podUID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" containerName="discovery" containerID="cri-o://890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788" gracePeriod=30 Oct 30 06:13:47.815947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.814881 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" event={"ID":"d87a41c6-002e-417e-a0aa-0e075df529e7","Type":"ContainerStarted","Data":"8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54"} Oct 30 06:13:47.815947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.814951 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" event={"ID":"d87a41c6-002e-417e-a0aa-0e075df529e7","Type":"ContainerStarted","Data":"24bfca9b0be65e771f6d63d2299d43538b549a352cea8a9a1d91f23ad14ee980"} Oct 30 06:13:47.833947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:47.830341 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:48.149617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.149598 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 06:13:48.226042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226015 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.226378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226355 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.226540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226527 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.226660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226649 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.226753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226743 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.226919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226903 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.226990 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.226961 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap\") pod \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\" (UID: \"49638c53-13ed-4a7e-8dc5-c5942c9117fe\") " Oct 30 06:13:48.227682 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.227652 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:48.229264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.229229 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:48.229558 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.229538 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:48.229695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.229677 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:48.229762 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.229737 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:48.229968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.229954 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:48.230462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.230444 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk" pod "49638c53-13ed-4a7e-8dc5-c5942c9117fe" (UID: "49638c53-13ed-4a7e-8dc5-c5942c9117fe"). InnerVolumeSpecName "kube-api-access-q5ckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:48.247443 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.247398 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podStartSLOduration=2.24738619 podStartE2EDuration="2.24738619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:47.893530391 +0000 UTC m=+2482.001569425" watchObservedRunningTime="2026-10-30 06:13:48.24738619 +0000 UTC m=+2482.355425222" Oct 30 06:13:48.329397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329364 5980 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-kubeconfig\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.329397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329391 5980 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-dns-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.329397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329402 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.329642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329410 5980 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/49638c53-13ed-4a7e-8dc5-c5942c9117fe-local-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.329642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329419 5980 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/49638c53-13ed-4a7e-8dc5-c5942c9117fe-istio-csr-ca-configmap\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.329642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329428 5980 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/49638c53-13ed-4a7e-8dc5-c5942c9117fe-cacerts\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.329642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.329435 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ckk\" (UniqueName: \"kubernetes.io/projected/49638c53-13ed-4a7e-8dc5-c5942c9117fe-kube-api-access-q5ckk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:13:48.431912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.431826 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 30 06:13:48.431912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.431872 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 30 06:13:48.823526 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.823495 5980 generic.go:356] "Generic (PLEG): container finished" podID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" containerID="890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788" exitCode=0 Oct 30 06:13:48.823996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.823571 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" Oct 30 06:13:48.823996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.823608 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" event={"ID":"49638c53-13ed-4a7e-8dc5-c5942c9117fe","Type":"ContainerDied","Data":"890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788"} Oct 30 06:13:48.823996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.823668 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk" event={"ID":"49638c53-13ed-4a7e-8dc5-c5942c9117fe","Type":"ContainerDied","Data":"1a7650f31b70e9186dd57628181c62acd1817edea52d29afa87fd4eb83017be4"} Oct 30 06:13:48.823996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.823696 5980 scope.go:122] "RemoveContainer" containerID="890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788" Oct 30 06:13:48.831291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.831268 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:13:48.831431 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.831322 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 06:13:48.838363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.838339 5980 scope.go:122] "RemoveContainer" containerID="890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788" Oct 30 06:13:48.838685 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:13:48.838665 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788\": container with ID starting with 890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788 not found: ID does not exist" containerID="890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788" Oct 30 06:13:48.838735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.838694 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788"} err="failed to get container status \"890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788\": rpc error: code = NotFound desc = could not find container \"890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788\": container with ID starting with 890225ad1daa38262b3360b738e928f3ed58aa79f6a7be8329358170c74c1788 not found: ID does not exist" Oct 30 06:13:48.883953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.883922 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk"] Oct 30 06:13:48.891667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:48.891627 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xfmtk"] Oct 30 06:13:49.436505 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:49.436476 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:13:49.436976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:49.436962 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:13:49.437813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:49.437796 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:13:49.830450 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:49.830416 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:13:49.830841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:49.830507 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 06:13:50.016967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:50.016935 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" path="/var/lib/kubelet/pods/49638c53-13ed-4a7e-8dc5-c5942c9117fe/volumes" Oct 30 06:13:50.506606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:50.506580 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:13:50.830352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:50.830323 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:13:50.830520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:50.830375 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 06:13:51.830579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:51.830545 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:13:51.831071 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:51.830600 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 06:13:52.830411 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:52.830379 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:13:52.830600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:52.830435 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 06:13:53.830664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:53.830636 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 06:13:53.831063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:53.830686 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 06:13:54.720736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:54.720708 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:13:54.955261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:54.955229 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:54.956180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:54.956156 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:54.957252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:54.957231 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:13:55.727231 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:55.727205 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:13:59.495950 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.495916 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb"] Oct 30 06:13:59.496655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.496637 5980 cpu_manager.go:414] "containerMap: removing container" podUID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" containerName="discovery" Oct 30 06:13:59.496722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.496657 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" containerName="discovery" Oct 30 06:13:59.496875 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.496848 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="49638c53-13ed-4a7e-8dc5-c5942c9117fe" containerName="discovery" Oct 30 06:13:59.501414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.501395 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.517629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.517608 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-f22a52-openshift-default-dockercfg-fmtw2\"" Oct 30 06:13:59.537207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.537172 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb"] Oct 30 06:13:59.649038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649008 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649063 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649150 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649233 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649266 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649389 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649450 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649417 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649505 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz67\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.649557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.649545 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750203 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750234 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750279 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz67\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750321 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750370 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750430 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750456 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750615 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750632 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750683 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750694 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.750913 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.750879 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.751175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.751154 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.751530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.751507 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.752758 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.752736 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.753133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.753084 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.775338 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.775313 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.775450 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.775399 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz67\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:13:59.816312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:13:59.816278 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:14:00.171153 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:00.171102 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ba055a_3427_42e5_b30a_9c1d89dcf790.slice/crio-29f45ffe79dbd71134dae0471ff33dcb005d9dbccb06afb380a6e86e0483b564 WatchSource:0}: Error finding container 29f45ffe79dbd71134dae0471ff33dcb005d9dbccb06afb380a6e86e0483b564: Status 404 returned error can't find the container with id 29f45ffe79dbd71134dae0471ff33dcb005d9dbccb06afb380a6e86e0483b564 Oct 30 06:14:00.172096 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:00.172067 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb"] Oct 30 06:14:00.174194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:00.174161 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:00.174266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:00.174230 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:00.174266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:00.174259 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:00.945087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:00.945045 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" event={"ID":"d5ba055a-3427-42e5-b30a-9c1d89dcf790","Type":"ContainerStarted","Data":"a44e9bf9a8be95eb1a8f759d0a12696431cfaa6c13ec701257898fd7cb312723"} Oct 30 06:14:00.945537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:00.945149 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" event={"ID":"d5ba055a-3427-42e5-b30a-9c1d89dcf790","Type":"ContainerStarted","Data":"29f45ffe79dbd71134dae0471ff33dcb005d9dbccb06afb380a6e86e0483b564"} Oct 30 06:14:01.041424 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.041377 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podStartSLOduration=2.041361771 podStartE2EDuration="2.041361771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:01.026333219 +0000 UTC m=+2495.134372251" watchObservedRunningTime="2026-10-30 06:14:01.041361771 +0000 UTC m=+2495.149400803" Oct 30 06:14:01.557609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.557576 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc"] Oct 30 06:14:01.563365 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.563336 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.594183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.593803 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-24db22-openshift-default-dockercfg-75mbl\"" Oct 30 06:14:01.605131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.597435 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc"] Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659292 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659341 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659428 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5x9\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659618 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659646 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659700 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659733 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659869 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.660135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.659913 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.761321 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.761286 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.761612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.761593 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.761779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.761763 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.761896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.761881 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.762047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.761759 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.768141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.762052 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5x9\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.772137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.768511 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.784136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.772328 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.784489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.784455 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.793229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.793195 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.805132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.784631 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.805850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.762378 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.805992 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.788655 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.806096 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.762097 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.809637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.809540 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.810426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.810407 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.810595 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.810455 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5x9\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.811285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.811252 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:01.817243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.816611 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:14:01.818197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.817922 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 06:14:01.818197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:01.817988 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 06:14:02.054696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.054665 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:02.499903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.499866 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc"] Oct 30 06:14:02.502208 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:02.502170 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a18b6a_88bf_43fd_b7a8_abae1840013d.slice/crio-578f29ddd8effaa0ecefaf3132ca5588194bcf597f5ea038a621d2c6412efbfd WatchSource:0}: Error finding container 578f29ddd8effaa0ecefaf3132ca5588194bcf597f5ea038a621d2c6412efbfd: Status 404 returned error can't find the container with id 578f29ddd8effaa0ecefaf3132ca5588194bcf597f5ea038a621d2c6412efbfd Oct 30 06:14:02.505292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.505261 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:02.505394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.505327 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:02.505394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.505357 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:02.817374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.817295 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 06:14:02.817374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.817362 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 06:14:02.966135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.963858 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" event={"ID":"26a18b6a-88bf-43fd-b7a8-abae1840013d","Type":"ContainerStarted","Data":"e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b"} Oct 30 06:14:02.966135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:02.963916 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" event={"ID":"26a18b6a-88bf-43fd-b7a8-abae1840013d","Type":"ContainerStarted","Data":"578f29ddd8effaa0ecefaf3132ca5588194bcf597f5ea038a621d2c6412efbfd"} Oct 30 06:14:03.020937 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:03.020884 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podStartSLOduration=2.020865786 podStartE2EDuration="2.020865786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:03.019014603 +0000 UTC m=+2497.127053636" watchObservedRunningTime="2026-10-30 06:14:03.020865786 +0000 UTC m=+2497.128904820" Oct 30 06:14:03.055686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:03.055651 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:03.057648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:03.057626 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:03.057742 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:03.057673 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:03.820133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:03.819583 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 06:14:03.820133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:03.819648 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 06:14:04.056358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:04.056330 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:04.056744 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:04.056380 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:04.830893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:04.830812 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 06:14:04.831215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:04.831181 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 06:14:05.055027 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:05.054996 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:05.055325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:05.055299 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:05.820172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:05.820137 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 06:14:05.820740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:05.820703 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 06:14:06.055852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:06.055740 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:06.055852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:06.055811 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:06.817567 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:06.817094 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 06:14:06.817567 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:06.817174 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 06:14:07.058514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:07.058479 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:07.059064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:07.059033 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:08.055399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:08.055364 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:08.055778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:08.055736 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:08.126812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:08.126783 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:14:08.127829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:08.127805 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:14:08.132156 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:08.132132 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:14:09.055295 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:09.055260 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 06:14:09.055446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:09.055307 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 06:14:10.060959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:10.060921 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:10.061480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:10.061331 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:10.062144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:10.062091 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:14:11.445806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.445771 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv"] Oct 30 06:14:11.451252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.451226 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.474760 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.474719 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv"] Oct 30 06:14:11.484084 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.484053 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f980e\"/\"default-dockercfg-wvlht\"" Oct 30 06:14:11.484354 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.484202 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f980e\"/\"auth-test-model-0f980e-kserve-self-signed-certs\"" Oct 30 06:14:11.488903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.488876 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f980e\"/\"kube-root-ca.crt\"" Oct 30 06:14:11.489036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.488914 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-0f980e\"/\"openshift-service-ca.crt\"" Oct 30 06:14:11.490698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.490677 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.490831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.490743 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.490831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.490773 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.490941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.490848 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.491036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.490996 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.491207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.491094 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.592364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.592322 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.592544 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.592421 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.592544 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.592503 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.592695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.592544 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.592695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.592649 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.592826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.592739 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.593500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.593474 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.594054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.594034 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.594417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.594398 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.596398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.596379 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.596820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.596801 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.619871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.619840 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2\") pod \"auth-test-model-0f980e-kserve-7686f99945-rlrbv\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:11.794352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:11.794204 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:12.216860 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:12.216827 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20858ec4_dbca_43b1_adab_4e6d3fb4aacc.slice/crio-8be0d2e1375b15519f9f18f9d43fffe7d6e191fccc8f70bfac3ea89cbcfb8a39 WatchSource:0}: Error finding container 8be0d2e1375b15519f9f18f9d43fffe7d6e191fccc8f70bfac3ea89cbcfb8a39: Status 404 returned error can't find the container with id 8be0d2e1375b15519f9f18f9d43fffe7d6e191fccc8f70bfac3ea89cbcfb8a39 Oct 30 06:14:12.217777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:12.217752 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv"] Oct 30 06:14:13.094900 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:14:13.094862 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:14:13.202433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:13.202390 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" event={"ID":"20858ec4-dbca-43b1-adab-4e6d3fb4aacc","Type":"ContainerStarted","Data":"faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65"} Oct 30 06:14:13.204011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:13.203979 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" event={"ID":"20858ec4-dbca-43b1-adab-4e6d3fb4aacc","Type":"ContainerStarted","Data":"8be0d2e1375b15519f9f18f9d43fffe7d6e191fccc8f70bfac3ea89cbcfb8a39"} Oct 30 06:14:13.255317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:13.255257 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" podStartSLOduration=2.255238111 podStartE2EDuration="2.255238111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:13.252943889 +0000 UTC m=+2507.360982932" watchObservedRunningTime="2026-10-30 06:14:13.255238111 +0000 UTC m=+2507.363277145" Oct 30 06:14:13.794955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:13.794913 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:21.392789 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.392744 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw"] Oct 30 06:14:21.399306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.399279 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.423487 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.423440 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw"] Oct 30 06:14:21.431987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.431960 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2d4576-openshift-default-dockercfg-bfgcw\"" Oct 30 06:14:21.485655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.485624 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.485862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.485687 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.485942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.485882 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.485942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.485930 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.486058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.486043 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2rg\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.486095 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.486078 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.486189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.486137 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.486238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.486213 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.486302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.486284 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.593306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.593270 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.594585 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.594559 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.594780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.594762 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.594985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.594969 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2rg\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.595143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.595128 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.595274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.595259 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.595425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.595411 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.595615 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.595601 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.595812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.595796 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.610323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.593917 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.610487 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.596835 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.610579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.597052 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.610679 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.597407 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.610766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.597642 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.642388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.632793 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2rg\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.642388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.642067 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.663200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.660704 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.663200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.662962 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo\") pod \"e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.721522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.720387 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:21.794680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.794625 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:21.823225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:21.821426 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:22.112218 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.112191 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw"] Oct 30 06:14:22.112812 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:22.112782 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb5598b_7e14_4635_be7a_8d80130a25ae.slice/crio-f4a56ce82c93a209c265380dd190cecd050483fbd40c27f73836e20303bac911 WatchSource:0}: Error finding container f4a56ce82c93a209c265380dd190cecd050483fbd40c27f73836e20303bac911: Status 404 returned error can't find the container with id f4a56ce82c93a209c265380dd190cecd050483fbd40c27f73836e20303bac911 Oct 30 06:14:22.115801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.115764 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:22.115904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.115830 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:22.115904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.115861 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:22.334410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.334367 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" event={"ID":"efb5598b-7e14-4635-be7a-8d80130a25ae","Type":"ContainerStarted","Data":"f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d"} Oct 30 06:14:22.334655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.334444 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" event={"ID":"efb5598b-7e14-4635-be7a-8d80130a25ae","Type":"ContainerStarted","Data":"f4a56ce82c93a209c265380dd190cecd050483fbd40c27f73836e20303bac911"} Oct 30 06:14:22.346854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.346829 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:14:22.387027 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.386924 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podStartSLOduration=1.386909298 podStartE2EDuration="1.386909298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:22.383223696 +0000 UTC m=+2516.491262755" watchObservedRunningTime="2026-10-30 06:14:22.386909298 +0000 UTC m=+2516.494948330" Oct 30 06:14:22.722994 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:22.722895 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:23.727868 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:23.727839 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:24.352380 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:24.352347 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:24.353299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:24.353272 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:14:26.791320 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.791286 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl"] Oct 30 06:14:26.796021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.795982 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:26.815606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.815576 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl"] Oct 30 06:14:26.824412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.824386 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d4576\"" Oct 30 06:14:26.824629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.824526 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d4576-dockercfg-lfbl9\"" Oct 30 06:14:26.972178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.964484 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqsfm\" (UniqueName: \"kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:26.972178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:26.964608 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:27.084516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:27.084479 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:27.105192 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:27.099777 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqsfm\" (UniqueName: \"kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:27.115132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:27.099052 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:27.142249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:27.141879 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqsfm\" (UniqueName: \"kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:27.413038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:27.412949 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:27.730670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:27.730641 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl"] Oct 30 06:14:27.730880 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:27.730857 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9f14a6_2a69_43f8_a8fc_ef9ec6ecb3da.slice/crio-0f2e074f1482573abdd47d82a07da6cb640a299bbe5ec2a29fa9d16168462ef9 WatchSource:0}: Error finding container 0f2e074f1482573abdd47d82a07da6cb640a299bbe5ec2a29fa9d16168462ef9: Status 404 returned error can't find the container with id 0f2e074f1482573abdd47d82a07da6cb640a299bbe5ec2a29fa9d16168462ef9 Oct 30 06:14:28.381631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:28.381592 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" event={"ID":"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da","Type":"ContainerStarted","Data":"a31ebd3860144506d23ac7af24195efcadd4c56fdcdf029009d529a674447372"} Oct 30 06:14:28.382031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:28.381675 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" event={"ID":"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da","Type":"ContainerStarted","Data":"0f2e074f1482573abdd47d82a07da6cb640a299bbe5ec2a29fa9d16168462ef9"} Oct 30 06:14:28.382031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:28.381705 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:28.438050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:28.437996 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" podStartSLOduration=2.437982298 podStartE2EDuration="2.437982298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:28.434447188 +0000 UTC m=+2522.542486220" watchObservedRunningTime="2026-10-30 06:14:28.437982298 +0000 UTC m=+2522.546021368" Oct 30 06:14:39.392903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:39.392811 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:14:43.101744 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:14:43.101709 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:14:50.152155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.140163 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2"] Oct 30 06:14:50.210191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.206427 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2"] Oct 30 06:14:50.210191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.206574 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.225559 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.225517 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf87e1\"" Oct 30 06:14:50.239968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.239944 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf87e1-dockercfg-64bjr\"" Oct 30 06:14:50.447040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.446940 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.447376 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.447352 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jtp\" (UniqueName: \"kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.548968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.548939 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.548968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.548978 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jtp\" (UniqueName: \"kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.549634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.549614 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.578345 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.578315 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jtp\" (UniqueName: \"kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:50.837617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:50.837591 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:51.159011 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:51.158979 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb45ba3_169c_4e79_ba7b_e3400c307aae.slice/crio-a3cf20673bd1d48b25a4587529ec1875d603854e50179bc611033fd8ab957e01 WatchSource:0}: Error finding container a3cf20673bd1d48b25a4587529ec1875d603854e50179bc611033fd8ab957e01: Status 404 returned error can't find the container with id a3cf20673bd1d48b25a4587529ec1875d603854e50179bc611033fd8ab957e01 Oct 30 06:14:51.159582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:51.159557 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2"] Oct 30 06:14:51.574542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:51.574505 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" event={"ID":"3cb45ba3-169c-4e79-ba7b-e3400c307aae","Type":"ContainerStarted","Data":"c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a"} Oct 30 06:14:51.574730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:51.574572 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" event={"ID":"3cb45ba3-169c-4e79-ba7b-e3400c307aae","Type":"ContainerStarted","Data":"a3cf20673bd1d48b25a4587529ec1875d603854e50179bc611033fd8ab957e01"} Oct 30 06:14:51.574730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:51.574605 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:14:51.626198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:51.625657 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" podStartSLOduration=1.625638591 podStartE2EDuration="1.625638591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:51.620378839 +0000 UTC m=+2545.728417873" watchObservedRunningTime="2026-10-30 06:14:51.625638591 +0000 UTC m=+2545.733677624" Oct 30 06:14:58.536264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.536233 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn"] Oct 30 06:14:58.541097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.541075 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.596330 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.596297 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f6302c-openshift-default-dockercfg-tplk9\"" Oct 30 06:14:58.610538 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.610499 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn"] Oct 30 06:14:58.626302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626268 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626313 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrzq\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626393 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626538 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626590 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626627 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626741 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626863 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.626966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.626946 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.737534 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.737606 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrzq\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.737651 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.737762 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.737793 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.737828 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.738359 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.759131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.738617 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.765668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.765259 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.782388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.765932 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.783536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.782677 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.784961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.784936 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.785381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.785358 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.790182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.786097 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.835135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.829267 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.854454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.854399 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.857687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.857651 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:58.915360 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:58.898831 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrzq\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq\") pod \"e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:59.161041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.161007 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:14:59.524758 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:14:59.524731 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc2dc0b_ba2c_4756_9190_d93b80a9cda0.slice/crio-c9e412d1d9a1e541e17a110bbf3678b414b10735fd36a2288b7a640e46ce8e78 WatchSource:0}: Error finding container c9e412d1d9a1e541e17a110bbf3678b414b10735fd36a2288b7a640e46ce8e78: Status 404 returned error can't find the container with id c9e412d1d9a1e541e17a110bbf3678b414b10735fd36a2288b7a640e46ce8e78 Oct 30 06:14:59.525747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.525723 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn"] Oct 30 06:14:59.527597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.527565 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:59.527666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.527632 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:59.527666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.527662 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:59.909835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.909795 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" event={"ID":"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0","Type":"ContainerStarted","Data":"8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e"} Oct 30 06:14:59.910251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:14:59.909861 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" event={"ID":"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0","Type":"ContainerStarted","Data":"c9e412d1d9a1e541e17a110bbf3678b414b10735fd36a2288b7a640e46ce8e78"} Oct 30 06:15:00.162101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.161339 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:15:00.163447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.163414 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:15:00.163609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.163466 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:15:00.237574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.233422 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podStartSLOduration=2.233395686 podStartE2EDuration="2.233395686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:00.04886379 +0000 UTC m=+2554.156902820" watchObservedRunningTime="2026-10-30 06:15:00.233395686 +0000 UTC m=+2554.341434718" Oct 30 06:15:00.237574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.236550 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk"] Oct 30 06:15:00.289557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.289521 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.293039 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.292065 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw"] Oct 30 06:15:00.298726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.297513 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.298726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.298252 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp"] Oct 30 06:15:00.314403 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.314256 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk"] Oct 30 06:15:00.314620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.314607 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q"] Oct 30 06:15:00.321351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.316251 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.347430 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.347189 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw"] Oct 30 06:15:00.347430 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.347383 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.361148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.360089 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp"] Oct 30 06:15:00.361585 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.361439 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q"] Oct 30 06:15:00.370815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.369389 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv"] Oct 30 06:15:00.382847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.382816 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:15:00.389659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.389638 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w"] Oct 30 06:15:00.398127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.390730 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.419048 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.419014 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv"] Oct 30 06:15:00.423406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.420406 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb"] Oct 30 06:15:00.426763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.426736 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.440163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.436995 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkpj\" (UniqueName: \"kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.440163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.437071 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.440163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.437148 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdm67\" (UniqueName: \"kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.440163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.437186 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.440163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.437292 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.440163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.437496 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcsqj\" (UniqueName: \"kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.463047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.463019 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4"] Oct 30 06:15:00.466481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.463162 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.480058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.480035 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb"] Oct 30 06:15:00.480504 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.480396 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.480708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.480696 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w"] Oct 30 06:15:00.485997 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.485979 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4"] Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539561 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539612 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539666 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdm67\" (UniqueName: \"kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539706 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539801 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-tlp2q\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539844 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.539855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539884 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv4nn\" (UniqueName: \"kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn\") pod \"maas-api-key-cleanup-29889015-tlp2q\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.540410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.539937 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.540410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.540089 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcsqj\" (UniqueName: \"kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.540410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.540158 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cqv\" (UniqueName: \"kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.540410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.540198 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mtn\" (UniqueName: \"kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.540410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.540319 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkpj\" (UniqueName: \"kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.541637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.541557 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.542234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.542194 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.542234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.542215 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.568570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.568540 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkpj\" (UniqueName: \"kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.569055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.569034 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdm67\" (UniqueName: \"kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.577091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.577064 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcsqj\" (UniqueName: \"kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.642015 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.641986 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.642174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642027 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.642237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642181 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cqv\" (UniqueName: \"kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.642237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642213 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mtn\" (UniqueName: \"kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.642359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642290 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.642359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642328 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.642467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642372 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vgn\" (UniqueName: \"kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.642467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642390 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2knq\" (UniqueName: \"kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.642467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642454 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-tlp2q\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.642970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642490 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv4nn\" (UniqueName: \"kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn\") pod \"maas-api-key-cleanup-29889015-tlp2q\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.642970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.642741 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.643171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.643150 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.643326 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.643304 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889015-tlp2q\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.685834 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.685763 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mtn\" (UniqueName: \"kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn\") pod \"maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.685996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.685971 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv4nn\" (UniqueName: \"kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn\") pod \"maas-api-key-cleanup-29889015-tlp2q\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.686458 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.686433 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cqv\" (UniqueName: \"kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:00.721082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.721058 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:00.744759 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.744731 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.744899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.744855 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vgn\" (UniqueName: \"kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.744899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.744889 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2knq\" (UniqueName: \"kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.745055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.745039 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.745485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.745460 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.745770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.745745 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.765227 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.765191 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:00.772724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.772610 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2knq\" (UniqueName: \"kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.774245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.774199 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vgn\" (UniqueName: \"kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.799147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.785089 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:00.820363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.803992 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:00.820363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.811758 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:00.837207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.835379 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:00.852236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.849516 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:00.852236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:00.849937 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:01.091282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.082525 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk"] Oct 30 06:15:01.091913 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:15:01.091886 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae284910_6e7d_419b_ad60_814f27522558.slice/crio-54db3d3722ddd8d11bc12c5d4f05dd8d17cc714c23d9ad500467fe0bac2dbc33 WatchSource:0}: Error finding container 54db3d3722ddd8d11bc12c5d4f05dd8d17cc714c23d9ad500467fe0bac2dbc33: Status 404 returned error can't find the container with id 54db3d3722ddd8d11bc12c5d4f05dd8d17cc714c23d9ad500467fe0bac2dbc33 Oct 30 06:15:01.173760 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.173420 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:15:01.385169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.385133 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw"] Oct 30 06:15:01.395755 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.395724 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp"] Oct 30 06:15:01.401941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.401890 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q"] Oct 30 06:15:01.404172 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:15:01.404018 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ade289_2392_4add_800c_9070a5a965ac.slice/crio-3ecf5390889c7f7f779db1bb21a3575c3cfdd7c208c8d99469b3ca8ccf279883 WatchSource:0}: Error finding container 3ecf5390889c7f7f779db1bb21a3575c3cfdd7c208c8d99469b3ca8ccf279883: Status 404 returned error can't find the container with id 3ecf5390889c7f7f779db1bb21a3575c3cfdd7c208c8d99469b3ca8ccf279883 Oct 30 06:15:01.474703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.474606 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv"] Oct 30 06:15:01.509215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.508238 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w"] Oct 30 06:15:01.512361 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.512335 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb"] Oct 30 06:15:01.515857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:01.515834 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4"] Oct 30 06:15:01.588609 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:15:01.588432 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56bf106_551f_46ec_b4ea_c8f6890dd29b.slice/crio-b1b6f810304d078095271363098ab79d3153ca6933bb088faaafb28bcc96ec15 WatchSource:0}: Error finding container b1b6f810304d078095271363098ab79d3153ca6933bb088faaafb28bcc96ec15: Status 404 returned error can't find the container with id b1b6f810304d078095271363098ab79d3153ca6933bb088faaafb28bcc96ec15 Oct 30 06:15:02.133353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.133260 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" event={"ID":"c3ade289-2392-4add-800c-9070a5a965ac","Type":"ContainerStarted","Data":"3ecf5390889c7f7f779db1bb21a3575c3cfdd7c208c8d99469b3ca8ccf279883"} Oct 30 06:15:02.133353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.133321 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" event={"ID":"38e8d144-5ea8-406c-bed8-c85a44336b68","Type":"ContainerStarted","Data":"79a0a8061f8d378de540a086598388ab583643f9b82853148b22305d4fa0fe14"} Oct 30 06:15:02.135715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.135409 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" event={"ID":"732c537b-6fa6-40db-aec8-5720bd35f93d","Type":"ContainerStarted","Data":"20ab3f9c7605316b1e5a2b962a141732dd1419197df504f54b24c706e59fd816"} Oct 30 06:15:02.136845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.136820 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" event={"ID":"a733495c-f173-4a82-b3a2-cb6bf48aa83d","Type":"ContainerStarted","Data":"874af00facb0116a0a999d10d80185c952d84b0190d28b86152c3c6e847076b4"} Oct 30 06:15:02.138507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.138484 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" event={"ID":"f56bf106-551f-46ec-b4ea-c8f6890dd29b","Type":"ContainerStarted","Data":"b1b6f810304d078095271363098ab79d3153ca6933bb088faaafb28bcc96ec15"} Oct 30 06:15:02.140157 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.140068 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerStarted","Data":"43ebc7620223d6ebe8d5648639273b3a0e9b20e73e1ac80ebf4ed367c7884513"} Oct 30 06:15:02.140157 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.140127 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerStarted","Data":"3ceae96a1dd56e2dd9464beea27ebbabf9a5236ef7c89ccb06023e74c81d778a"} Oct 30 06:15:02.141879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.141532 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerStarted","Data":"3de0400158cd94ac2f67213cd80dded19a8875df7161f0341a993f5add49a2e1"} Oct 30 06:15:02.141879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.141569 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerStarted","Data":"54db3d3722ddd8d11bc12c5d4f05dd8d17cc714c23d9ad500467fe0bac2dbc33"} Oct 30 06:15:02.143408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.143025 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" event={"ID":"fce43331-a061-418e-87ee-cb48df2fc345","Type":"ContainerStarted","Data":"4b2d104c0ca6bc3fe41e291ca403a7bd4fbec43d9345bd52cedb4e17df8a09d1"} Oct 30 06:15:02.143408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.143372 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:15:02.144329 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.144315 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:15:02.184382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.184292 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" podStartSLOduration=2.1842748260000002 podStartE2EDuration="2.184274826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.179539196 +0000 UTC m=+2556.287578228" watchObservedRunningTime="2026-10-30 06:15:02.184274826 +0000 UTC m=+2556.292313861" Oct 30 06:15:02.224073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.224030 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" podStartSLOduration=2.224017329 podStartE2EDuration="2.224017329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.221171718 +0000 UTC m=+2556.329210751" watchObservedRunningTime="2026-10-30 06:15:02.224017329 +0000 UTC m=+2556.332056414" Oct 30 06:15:02.330070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.330003 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" podStartSLOduration=2.329983527 podStartE2EDuration="2.329983527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.321309005 +0000 UTC m=+2556.429348037" watchObservedRunningTime="2026-10-30 06:15:02.329983527 +0000 UTC m=+2556.438022561" Oct 30 06:15:02.372182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.370401 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" podStartSLOduration=2.370381557 podStartE2EDuration="2.370381557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.368563136 +0000 UTC m=+2556.476602169" watchObservedRunningTime="2026-10-30 06:15:02.370381557 +0000 UTC m=+2556.478420587" Oct 30 06:15:02.430241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.430161 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" podStartSLOduration=2.430142123 podStartE2EDuration="2.430142123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.413565993 +0000 UTC m=+2556.521605040" watchObservedRunningTime="2026-10-30 06:15:02.430142123 +0000 UTC m=+2556.538181192" Oct 30 06:15:02.594217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:02.592368 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:15:03.154447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.154354 5980 generic.go:356] "Generic (PLEG): container finished" podID="25609759-f118-43eb-bbd2-3f42e12da38f" containerID="43ebc7620223d6ebe8d5648639273b3a0e9b20e73e1ac80ebf4ed367c7884513" exitCode=60 Oct 30 06:15:03.154890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.154452 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerDied","Data":"43ebc7620223d6ebe8d5648639273b3a0e9b20e73e1ac80ebf4ed367c7884513"} Oct 30 06:15:03.154890 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.154775 5980 scope.go:122] "RemoveContainer" containerID="43ebc7620223d6ebe8d5648639273b3a0e9b20e73e1ac80ebf4ed367c7884513" Oct 30 06:15:03.156519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.156490 5980 generic.go:356] "Generic (PLEG): container finished" podID="ae284910-6e7d-419b-ad60-814f27522558" containerID="3de0400158cd94ac2f67213cd80dded19a8875df7161f0341a993f5add49a2e1" exitCode=60 Oct 30 06:15:03.156665 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.156583 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerDied","Data":"3de0400158cd94ac2f67213cd80dded19a8875df7161f0341a993f5add49a2e1"} Oct 30 06:15:03.157269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.156882 5980 scope.go:122] "RemoveContainer" containerID="3de0400158cd94ac2f67213cd80dded19a8875df7161f0341a993f5add49a2e1" Oct 30 06:15:03.163626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.163596 5980 generic.go:356] "Generic (PLEG): container finished" podID="fce43331-a061-418e-87ee-cb48df2fc345" containerID="1fbbe3e5b4b3f57e6a4a0bacefe36650b440a4f1be6b336e2b5da4bca19d7726" exitCode=60 Oct 30 06:15:03.163925 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.163867 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" event={"ID":"fce43331-a061-418e-87ee-cb48df2fc345","Type":"ContainerDied","Data":"1fbbe3e5b4b3f57e6a4a0bacefe36650b440a4f1be6b336e2b5da4bca19d7726"} Oct 30 06:15:03.164551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.164534 5980 scope.go:122] "RemoveContainer" containerID="1fbbe3e5b4b3f57e6a4a0bacefe36650b440a4f1be6b336e2b5da4bca19d7726" Oct 30 06:15:03.166251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.166224 5980 generic.go:356] "Generic (PLEG): container finished" podID="c3ade289-2392-4add-800c-9070a5a965ac" containerID="3c0e1cb2c3e41a8a75208a4b37af60cc180ad579060bc2053860ba4aa2f87aa9" exitCode=60 Oct 30 06:15:03.166377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.166359 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" event={"ID":"c3ade289-2392-4add-800c-9070a5a965ac","Type":"ContainerDied","Data":"3c0e1cb2c3e41a8a75208a4b37af60cc180ad579060bc2053860ba4aa2f87aa9"} Oct 30 06:15:03.166565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.166531 5980 scope.go:122] "RemoveContainer" containerID="3c0e1cb2c3e41a8a75208a4b37af60cc180ad579060bc2053860ba4aa2f87aa9" Oct 30 06:15:03.170726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.170704 5980 generic.go:356] "Generic (PLEG): container finished" podID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerID="2807283b1e080bcd9a7148648a6a19275b6ad4a164e5ba1ee1302123f5bd3df3" exitCode=60 Oct 30 06:15:03.170823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.170774 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" event={"ID":"38e8d144-5ea8-406c-bed8-c85a44336b68","Type":"ContainerDied","Data":"2807283b1e080bcd9a7148648a6a19275b6ad4a164e5ba1ee1302123f5bd3df3"} Oct 30 06:15:03.172318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.171596 5980 scope.go:122] "RemoveContainer" containerID="2807283b1e080bcd9a7148648a6a19275b6ad4a164e5ba1ee1302123f5bd3df3" Oct 30 06:15:03.193240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.193209 5980 generic.go:356] "Generic (PLEG): container finished" podID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerID="75860317022c0815f66fddef3fd0c8a7555a496f5f730abfd1e72f65e7a829c4" exitCode=60 Oct 30 06:15:03.193412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.193393 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" event={"ID":"732c537b-6fa6-40db-aec8-5720bd35f93d","Type":"ContainerDied","Data":"75860317022c0815f66fddef3fd0c8a7555a496f5f730abfd1e72f65e7a829c4"} Oct 30 06:15:03.193860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.193839 5980 scope.go:122] "RemoveContainer" containerID="75860317022c0815f66fddef3fd0c8a7555a496f5f730abfd1e72f65e7a829c4" Oct 30 06:15:03.199917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.199893 5980 generic.go:356] "Generic (PLEG): container finished" podID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerID="647b59c3458c204523c5bb9811e38423365bd1fb5be03e16830da07c31f31ba6" exitCode=60 Oct 30 06:15:03.200480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.200321 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" event={"ID":"a733495c-f173-4a82-b3a2-cb6bf48aa83d","Type":"ContainerDied","Data":"647b59c3458c204523c5bb9811e38423365bd1fb5be03e16830da07c31f31ba6"} Oct 30 06:15:03.200572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.200386 5980 scope.go:122] "RemoveContainer" containerID="647b59c3458c204523c5bb9811e38423365bd1fb5be03e16830da07c31f31ba6" Oct 30 06:15:03.210676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.210647 5980 generic.go:356] "Generic (PLEG): container finished" podID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerID="abe389980a3066235bbb091586243c2ae6f3bb2b50efe84037df629f0e64464d" exitCode=60 Oct 30 06:15:03.211826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.211806 5980 scope.go:122] "RemoveContainer" containerID="abe389980a3066235bbb091586243c2ae6f3bb2b50efe84037df629f0e64464d" Oct 30 06:15:03.212239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:03.212209 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" event={"ID":"f56bf106-551f-46ec-b4ea-c8f6890dd29b","Type":"ContainerDied","Data":"abe389980a3066235bbb091586243c2ae6f3bb2b50efe84037df629f0e64464d"} Oct 30 06:15:04.223513 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.223459 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerStarted","Data":"9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd"} Oct 30 06:15:04.225533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.225505 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerStarted","Data":"780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313"} Oct 30 06:15:04.227594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.227571 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" event={"ID":"fce43331-a061-418e-87ee-cb48df2fc345","Type":"ContainerStarted","Data":"972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f"} Oct 30 06:15:04.231123 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.231082 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" event={"ID":"c3ade289-2392-4add-800c-9070a5a965ac","Type":"ContainerStarted","Data":"cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8"} Oct 30 06:15:04.233748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.233727 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" event={"ID":"38e8d144-5ea8-406c-bed8-c85a44336b68","Type":"ContainerStarted","Data":"57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2"} Oct 30 06:15:04.235912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.235867 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" event={"ID":"732c537b-6fa6-40db-aec8-5720bd35f93d","Type":"ContainerStarted","Data":"506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1"} Oct 30 06:15:04.240069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.240023 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" event={"ID":"a733495c-f173-4a82-b3a2-cb6bf48aa83d","Type":"ContainerStarted","Data":"b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f"} Oct 30 06:15:04.243245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.243223 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" event={"ID":"f56bf106-551f-46ec-b4ea-c8f6890dd29b","Type":"ContainerStarted","Data":"80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a"} Oct 30 06:15:04.329786 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.329735 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" podStartSLOduration=4.329717178 podStartE2EDuration="4.329717178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.324023242 +0000 UTC m=+2558.432062277" watchObservedRunningTime="2026-10-30 06:15:04.329717178 +0000 UTC m=+2558.437756211" Oct 30 06:15:04.378304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.378243 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" podStartSLOduration=4.378224641 podStartE2EDuration="4.378224641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.369456088 +0000 UTC m=+2558.477495123" watchObservedRunningTime="2026-10-30 06:15:04.378224641 +0000 UTC m=+2558.486263690" Oct 30 06:15:04.428202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.428171 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6"] Oct 30 06:15:04.470647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.470618 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6"] Oct 30 06:15:04.470817 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.470676 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.517499 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.516247 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f6302c\"" Oct 30 06:15:04.517879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.517833 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f6302c-dockercfg-wkq6s\"" Oct 30 06:15:04.597074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.597037 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg9kv\" (UniqueName: \"kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv\") pod \"payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.597388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.597192 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.732772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.730932 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.732772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.731322 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg9kv\" (UniqueName: \"kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv\") pod \"payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.732772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.732678 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.743179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.734643 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" podStartSLOduration=4.734625988 podStartE2EDuration="4.734625988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.727508442 +0000 UTC m=+2558.835547501" watchObservedRunningTime="2026-10-30 06:15:04.734625988 +0000 UTC m=+2558.842665022" Oct 30 06:15:04.782922 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.782650 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg9kv\" (UniqueName: \"kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv\") pod \"payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:04.892828 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:04.892795 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:05.254237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.254159 5980 generic.go:356] "Generic (PLEG): container finished" podID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerID="b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f" exitCode=60 Oct 30 06:15:05.254696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.254243 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" event={"ID":"a733495c-f173-4a82-b3a2-cb6bf48aa83d","Type":"ContainerDied","Data":"b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f"} Oct 30 06:15:05.254696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.254316 5980 scope.go:122] "RemoveContainer" containerID="647b59c3458c204523c5bb9811e38423365bd1fb5be03e16830da07c31f31ba6" Oct 30 06:15:05.254696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.254521 5980 scope.go:122] "RemoveContainer" containerID="b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f" Oct 30 06:15:05.254888 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.254866 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4_odh-ai-gateway-infra(a733495c-f173-4a82-b3a2-cb6bf48aa83d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" Oct 30 06:15:05.256533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.256510 5980 generic.go:356] "Generic (PLEG): container finished" podID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerID="80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a" exitCode=60 Oct 30 06:15:05.256636 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.256584 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" event={"ID":"f56bf106-551f-46ec-b4ea-c8f6890dd29b","Type":"ContainerDied","Data":"80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a"} Oct 30 06:15:05.256821 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.256806 5980 scope.go:122] "RemoveContainer" containerID="80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a" Oct 30 06:15:05.257074 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.257055 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb_odh-ai-gateway-infra(f56bf106-551f-46ec-b4ea-c8f6890dd29b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" Oct 30 06:15:05.258636 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.258616 5980 generic.go:356] "Generic (PLEG): container finished" podID="25609759-f118-43eb-bbd2-3f42e12da38f" containerID="9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd" exitCode=60 Oct 30 06:15:05.258747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.258685 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerDied","Data":"9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd"} Oct 30 06:15:05.258877 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.258862 5980 scope.go:122] "RemoveContainer" containerID="9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd" Oct 30 06:15:05.259146 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.259089 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp_odh-ai-gateway-infra(25609759-f118-43eb-bbd2-3f42e12da38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" Oct 30 06:15:05.260608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.260591 5980 generic.go:356] "Generic (PLEG): container finished" podID="ae284910-6e7d-419b-ad60-814f27522558" containerID="780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313" exitCode=60 Oct 30 06:15:05.260700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.260670 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerDied","Data":"780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313"} Oct 30 06:15:05.260846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.260831 5980 scope.go:122] "RemoveContainer" containerID="780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313" Oct 30 06:15:05.261065 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.261046 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk_odh-ai-gateway-infra(ae284910-6e7d-419b-ad60-814f27522558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" podUID="ae284910-6e7d-419b-ad60-814f27522558" Oct 30 06:15:05.262576 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.262555 5980 generic.go:356] "Generic (PLEG): container finished" podID="fce43331-a061-418e-87ee-cb48df2fc345" containerID="972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f" exitCode=60 Oct 30 06:15:05.262682 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.262637 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" event={"ID":"fce43331-a061-418e-87ee-cb48df2fc345","Type":"ContainerDied","Data":"972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f"} Oct 30 06:15:05.262856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.262841 5980 scope.go:122] "RemoveContainer" containerID="972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f" Oct 30 06:15:05.263176 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.263154 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w_odh-ai-gateway-infra(fce43331-a061-418e-87ee-cb48df2fc345)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" podUID="fce43331-a061-418e-87ee-cb48df2fc345" Oct 30 06:15:05.264769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.264746 5980 generic.go:356] "Generic (PLEG): container finished" podID="c3ade289-2392-4add-800c-9070a5a965ac" containerID="cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8" exitCode=60 Oct 30 06:15:05.264869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.264836 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" event={"ID":"c3ade289-2392-4add-800c-9070a5a965ac","Type":"ContainerDied","Data":"cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8"} Oct 30 06:15:05.265068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.265027 5980 scope.go:122] "RemoveContainer" containerID="cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8" Oct 30 06:15:05.265354 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.265319 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-tlp2q_odh-ai-gateway-infra(c3ade289-2392-4add-800c-9070a5a965ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" podUID="c3ade289-2392-4add-800c-9070a5a965ac" Oct 30 06:15:05.267025 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.267007 5980 generic.go:356] "Generic (PLEG): container finished" podID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerID="57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2" exitCode=60 Oct 30 06:15:05.267158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.267073 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" event={"ID":"38e8d144-5ea8-406c-bed8-c85a44336b68","Type":"ContainerDied","Data":"57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2"} Oct 30 06:15:05.267359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.267341 5980 scope.go:122] "RemoveContainer" containerID="57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2" Oct 30 06:15:05.267613 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.267592 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv_odh-ai-gateway-infra(38e8d144-5ea8-406c-bed8-c85a44336b68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" Oct 30 06:15:05.269504 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.269481 5980 generic.go:356] "Generic (PLEG): container finished" podID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerID="506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1" exitCode=60 Oct 30 06:15:05.269597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.269529 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" event={"ID":"732c537b-6fa6-40db-aec8-5720bd35f93d","Type":"ContainerDied","Data":"506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1"} Oct 30 06:15:05.269817 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.269800 5980 scope.go:122] "RemoveContainer" containerID="506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1" Oct 30 06:15:05.270086 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:05.270066 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw_odh-ai-gateway-infra(732c537b-6fa6-40db-aec8-5720bd35f93d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" Oct 30 06:15:05.292315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.292199 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:15:05.292315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.292251 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:15:05.292702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.292656 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6"] Oct 30 06:15:05.301706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.301685 5980 scope.go:122] "RemoveContainer" containerID="abe389980a3066235bbb091586243c2ae6f3bb2b50efe84037df629f0e64464d" Oct 30 06:15:05.355729 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.355705 5980 scope.go:122] "RemoveContainer" containerID="43ebc7620223d6ebe8d5648639273b3a0e9b20e73e1ac80ebf4ed367c7884513" Oct 30 06:15:05.403020 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.402996 5980 scope.go:122] "RemoveContainer" containerID="3de0400158cd94ac2f67213cd80dded19a8875df7161f0341a993f5add49a2e1" Oct 30 06:15:05.437568 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.437546 5980 scope.go:122] "RemoveContainer" containerID="1fbbe3e5b4b3f57e6a4a0bacefe36650b440a4f1be6b336e2b5da4bca19d7726" Oct 30 06:15:05.470386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.470362 5980 scope.go:122] "RemoveContainer" containerID="3c0e1cb2c3e41a8a75208a4b37af60cc180ad579060bc2053860ba4aa2f87aa9" Oct 30 06:15:05.506796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.506771 5980 scope.go:122] "RemoveContainer" containerID="2807283b1e080bcd9a7148648a6a19275b6ad4a164e5ba1ee1302123f5bd3df3" Oct 30 06:15:05.538714 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:05.538691 5980 scope.go:122] "RemoveContainer" containerID="75860317022c0815f66fddef3fd0c8a7555a496f5f730abfd1e72f65e7a829c4" Oct 30 06:15:06.284210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.280380 5980 scope.go:122] "RemoveContainer" containerID="9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd" Oct 30 06:15:06.284210 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.280779 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp_odh-ai-gateway-infra(25609759-f118-43eb-bbd2-3f42e12da38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" Oct 30 06:15:06.284210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.282365 5980 scope.go:122] "RemoveContainer" containerID="780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313" Oct 30 06:15:06.284210 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.282734 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk_odh-ai-gateway-infra(ae284910-6e7d-419b-ad60-814f27522558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" podUID="ae284910-6e7d-419b-ad60-814f27522558" Oct 30 06:15:06.284210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.284075 5980 scope.go:122] "RemoveContainer" containerID="972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f" Oct 30 06:15:06.285985 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.284384 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w_odh-ai-gateway-infra(fce43331-a061-418e-87ee-cb48df2fc345)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" podUID="fce43331-a061-418e-87ee-cb48df2fc345" Oct 30 06:15:06.285985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.285889 5980 scope.go:122] "RemoveContainer" containerID="cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8" Oct 30 06:15:06.286220 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.286194 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-tlp2q_odh-ai-gateway-infra(c3ade289-2392-4add-800c-9070a5a965ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" podUID="c3ade289-2392-4add-800c-9070a5a965ac" Oct 30 06:15:06.287767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.287748 5980 scope.go:122] "RemoveContainer" containerID="57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2" Oct 30 06:15:06.288000 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.287978 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv_odh-ai-gateway-infra(38e8d144-5ea8-406c-bed8-c85a44336b68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" Oct 30 06:15:06.289550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.289527 5980 scope.go:122] "RemoveContainer" containerID="506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1" Oct 30 06:15:06.289766 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.289740 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw_odh-ai-gateway-infra(732c537b-6fa6-40db-aec8-5720bd35f93d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" Oct 30 06:15:06.291023 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.290995 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" event={"ID":"1c651b7d-ee0a-43fe-a4e7-7ca798962171","Type":"ContainerStarted","Data":"05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2"} Oct 30 06:15:06.291155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.291087 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:06.291227 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.291156 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" event={"ID":"1c651b7d-ee0a-43fe-a4e7-7ca798962171","Type":"ContainerStarted","Data":"2a37efcf7af34787eea39c8a018178dbd5644a2efeab0cbe85510f71fd4f63cc"} Oct 30 06:15:06.293026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.292990 5980 scope.go:122] "RemoveContainer" containerID="b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f" Oct 30 06:15:06.293314 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.293284 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4_odh-ai-gateway-infra(a733495c-f173-4a82-b3a2-cb6bf48aa83d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" Oct 30 06:15:06.294711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.294693 5980 scope.go:122] "RemoveContainer" containerID="80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a" Oct 30 06:15:06.294926 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:06.294908 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb_odh-ai-gateway-infra(f56bf106-551f-46ec-b4ea-c8f6890dd29b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" Oct 30 06:15:06.689041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:06.687792 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" podStartSLOduration=2.687761382 podStartE2EDuration="2.687761382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:06.683088146 +0000 UTC m=+2560.791127179" watchObservedRunningTime="2026-10-30 06:15:06.687761382 +0000 UTC m=+2560.795800418" Oct 30 06:15:07.308959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.308874 5980 generic.go:356] "Generic (PLEG): container finished" podID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerID="2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4" exitCode=0 Oct 30 06:15:07.309404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.308948 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" event={"ID":"1a254368-d09a-4e4a-a6e3-2f2e1446c64a","Type":"ContainerDied","Data":"2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4"} Oct 30 06:15:07.309822 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.309801 5980 scope.go:122] "RemoveContainer" containerID="2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4" Oct 30 06:15:07.311981 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.311921 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:07.312125 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.312004 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:07.312125 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.312039 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:07.446901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.446870 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr"] Oct 30 06:15:07.499998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:07.499968 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k"] Oct 30 06:15:08.354127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:08.353877 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" event={"ID":"1a254368-d09a-4e4a-a6e3-2f2e1446c64a","Type":"ContainerStarted","Data":"fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2"} Oct 30 06:15:08.650359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:08.650279 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:15:08.650674 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:08.650655 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:15:08.650743 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:08.650703 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 06:15:09.650145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:09.650094 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:15:09.650547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:09.650175 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 06:15:10.649819 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:10.649790 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:15:10.650002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:10.649845 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 06:15:11.650145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:11.650102 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:15:11.650535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:11.650169 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 06:15:12.448058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.448006 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" podUID="723630fe-6645-486c-8e7c-264fa375ab6d" containerName="payload-processing" containerID="cri-o://640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a" gracePeriod=25 Oct 30 06:15:12.501036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.500987 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" podUID="8d877394-f587-4642-a895-75140b325d02" containerName="payload-pre-processing" containerID="cri-o://cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00" gracePeriod=25 Oct 30 06:15:12.649570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.649539 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:15:12.649735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.649585 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 06:15:12.900196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.896691 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:15:12.937388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.937356 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:15:12.937668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.937635 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume\") pod \"723630fe-6645-486c-8e7c-264fa375ab6d\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " Oct 30 06:15:12.937889 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.937868 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb\" (UniqueName: \"kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb\") pod \"723630fe-6645-486c-8e7c-264fa375ab6d\" (UID: \"723630fe-6645-486c-8e7c-264fa375ab6d\") " Oct 30 06:15:12.938044 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.938023 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume" pod "723630fe-6645-486c-8e7c-264fa375ab6d" (UID: "723630fe-6645-486c-8e7c-264fa375ab6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:12.938588 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.938567 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723630fe-6645-486c-8e7c-264fa375ab6d-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:12.941382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:12.940949 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb" pod "723630fe-6645-486c-8e7c-264fa375ab6d" (UID: "723630fe-6645-486c-8e7c-264fa375ab6d"). InnerVolumeSpecName "kube-api-access-ccgbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:13.039421 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.039324 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume\") pod \"8d877394-f587-4642-a895-75140b325d02\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " Oct 30 06:15:13.039606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.039459 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg\" (UniqueName: \"kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg\") pod \"8d877394-f587-4642-a895-75140b325d02\" (UID: \"8d877394-f587-4642-a895-75140b325d02\") " Oct 30 06:15:13.039767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.039736 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume" pod "8d877394-f587-4642-a895-75140b325d02" (UID: "8d877394-f587-4642-a895-75140b325d02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:13.040208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.040186 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccgbb\" (UniqueName: \"kubernetes.io/projected/723630fe-6645-486c-8e7c-264fa375ab6d-kube-api-access-ccgbb\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.040313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.040211 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d877394-f587-4642-a895-75140b325d02-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.042307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.042284 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg" pod "8d877394-f587-4642-a895-75140b325d02" (UID: "8d877394-f587-4642-a895-75140b325d02"). InnerVolumeSpecName "kube-api-access-mpsvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:13.124220 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:13.124184 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:15:13.142117 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.142078 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpsvg\" (UniqueName: \"kubernetes.io/projected/8d877394-f587-4642-a895-75140b325d02-kube-api-access-mpsvg\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.444233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.443653 5980 generic.go:356] "Generic (PLEG): container finished" podID="8d877394-f587-4642-a895-75140b325d02" containerID="cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00" exitCode=0 Oct 30 06:15:13.444233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.443732 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" event={"ID":"8d877394-f587-4642-a895-75140b325d02","Type":"ContainerDied","Data":"cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00"} Oct 30 06:15:13.444233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.443783 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" event={"ID":"8d877394-f587-4642-a895-75140b325d02","Type":"ContainerDied","Data":"4b9d3f3fea94df76077bf4fcdb7c0676f4f6db5fab581d29af7dc8170377eb70"} Oct 30 06:15:13.444233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.443814 5980 scope.go:122] "RemoveContainer" containerID="cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00" Oct 30 06:15:13.445620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.445284 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k" Oct 30 06:15:13.446014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.445984 5980 generic.go:356] "Generic (PLEG): container finished" podID="723630fe-6645-486c-8e7c-264fa375ab6d" containerID="640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a" exitCode=0 Oct 30 06:15:13.446138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.446053 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" Oct 30 06:15:13.446210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.446164 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" event={"ID":"723630fe-6645-486c-8e7c-264fa375ab6d","Type":"ContainerDied","Data":"640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a"} Oct 30 06:15:13.446281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.446212 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr" event={"ID":"723630fe-6645-486c-8e7c-264fa375ab6d","Type":"ContainerDied","Data":"209c7b120400f69134ffa5a2c6ccf91a5013aa271743a1e32c370c802911b1b2"} Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.514179 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j"] Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.514945 5980 cpu_manager.go:414] "containerMap: removing container" podUID="723630fe-6645-486c-8e7c-264fa375ab6d" containerName="payload-processing" Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.514960 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723630fe-6645-486c-8e7c-264fa375ab6d" containerName="payload-processing" Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.514975 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8d877394-f587-4642-a895-75140b325d02" containerName="payload-pre-processing" Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.514981 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d877394-f587-4642-a895-75140b325d02" containerName="payload-pre-processing" Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.515090 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="723630fe-6645-486c-8e7c-264fa375ab6d" containerName="payload-processing" Oct 30 06:15:13.516134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.515101 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d877394-f587-4642-a895-75140b325d02" containerName="payload-pre-processing" Oct 30 06:15:13.517909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.517618 5980 scope.go:122] "RemoveContainer" containerID="cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00" Oct 30 06:15:13.518060 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:13.517946 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00\": container with ID starting with cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00 not found: ID does not exist" containerID="cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00" Oct 30 06:15:13.518060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.517975 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00"} err="failed to get container status \"cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00\": rpc error: code = NotFound desc = could not find container \"cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00\": container with ID starting with cde27a8adfa3883f69650fe3d30c5711e728a756d6e389b65880856f4e68ae00 not found: ID does not exist" Oct 30 06:15:13.518060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.517993 5980 scope.go:122] "RemoveContainer" containerID="640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a" Oct 30 06:15:13.519607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.519586 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.538361 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.538340 5980 scope.go:122] "RemoveContainer" containerID="640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a" Oct 30 06:15:13.538751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.538732 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0f980e-openshift-default-dockercfg-sk7nx\"" Oct 30 06:15:13.539870 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:13.539837 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a\": container with ID starting with 640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a not found: ID does not exist" containerID="640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a" Oct 30 06:15:13.540165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.539880 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a"} err="failed to get container status \"640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a\": rpc error: code = NotFound desc = could not find container \"640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a\": container with ID starting with 640224610b3f01b8fda3e0a4b9470a7fa094aeaa4264011f921ed6992aa1070a not found: ID does not exist" Oct 30 06:15:13.540165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.540073 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j"] Oct 30 06:15:13.550274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.550248 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.550524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.550501 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.550724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.550703 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.550854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.550837 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.551679 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.551658 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.551870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.551849 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.552068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.552049 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzc9\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.552390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.552370 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.552526 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.552510 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.597197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.597161 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k"] Oct 30 06:15:13.602964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.602937 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-5766b8f84c-tjh5k"] Oct 30 06:15:13.619427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.619403 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr"] Oct 30 06:15:13.625193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.625163 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-55b47855ff-f9ncr"] Oct 30 06:15:13.650336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.650313 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 06:15:13.650457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.650365 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 06:15:13.654180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654158 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654328 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzc9\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654462 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654551 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654669 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654657 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654736 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654827 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.654962 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.654945 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.655083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.655068 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.655315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.655294 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.655476 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.655333 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.655572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.655149 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.655659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.655487 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.655746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.655674 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.658098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.657973 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.658226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.658209 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.691863 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.691834 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.697858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.697789 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzc9\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:13.852596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:13.852553 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:14.022667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.022636 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723630fe-6645-486c-8e7c-264fa375ab6d" path="/var/lib/kubelet/pods/723630fe-6645-486c-8e7c-264fa375ab6d/volumes" Oct 30 06:15:14.023262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.023243 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d877394-f587-4642-a895-75140b325d02" path="/var/lib/kubelet/pods/8d877394-f587-4642-a895-75140b325d02/volumes" Oct 30 06:15:14.217178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.216224 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j"] Oct 30 06:15:14.223633 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:15:14.223597 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4018a797_c102_4cbb_9e8d_8ff7a87098d6.slice/crio-f91e2064a312080fedd7fff94a9cd7fa3ac76e7594e6d492e731d3f4bf9c1148 WatchSource:0}: Error finding container f91e2064a312080fedd7fff94a9cd7fa3ac76e7594e6d492e731d3f4bf9c1148: Status 404 returned error can't find the container with id f91e2064a312080fedd7fff94a9cd7fa3ac76e7594e6d492e731d3f4bf9c1148 Oct 30 06:15:14.226729 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.226694 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:14.226845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.226776 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:14.226845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.226807 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:14.457293 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.457256 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" event={"ID":"4018a797-c102-4cbb-9e8d-8ff7a87098d6","Type":"ContainerStarted","Data":"e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1"} Oct 30 06:15:14.457519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.457322 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" event={"ID":"4018a797-c102-4cbb-9e8d-8ff7a87098d6","Type":"ContainerStarted","Data":"f91e2064a312080fedd7fff94a9cd7fa3ac76e7594e6d492e731d3f4bf9c1148"} Oct 30 06:15:14.518679 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.518515 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podStartSLOduration=1.5184970469999999 podStartE2EDuration="1.518497047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:14.517834549 +0000 UTC m=+2568.625873585" watchObservedRunningTime="2026-10-30 06:15:14.518497047 +0000 UTC m=+2568.626536083" Oct 30 06:15:14.655441 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.655393 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:15:14.655831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.655810 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:15:14.656773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.656748 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:15:14.853440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:14.853403 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:15.007201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:15.007170 5980 scope.go:122] "RemoveContainer" containerID="cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8" Oct 30 06:15:15.854078 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:15.854050 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 06:15:15.854521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:15.854101 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 06:15:16.514138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:16.514093 5980 generic.go:356] "Generic (PLEG): container finished" podID="c3ade289-2392-4add-800c-9070a5a965ac" containerID="1aa5ed037976b0b59ab80cf7926ffa2d822c1d5578fb8afda20a2a93b349b208" exitCode=60 Oct 30 06:15:16.514350 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:16.514140 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" event={"ID":"c3ade289-2392-4add-800c-9070a5a965ac","Type":"ContainerDied","Data":"1aa5ed037976b0b59ab80cf7926ffa2d822c1d5578fb8afda20a2a93b349b208"} Oct 30 06:15:16.514350 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:16.514212 5980 scope.go:122] "RemoveContainer" containerID="cf60bc5903664fcf7e2420c2909732a277f39c1b20e11f22301567f2383535f8" Oct 30 06:15:16.514954 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:16.514930 5980 scope.go:122] "RemoveContainer" containerID="1aa5ed037976b0b59ab80cf7926ffa2d822c1d5578fb8afda20a2a93b349b208" Oct 30 06:15:16.515636 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:16.515295 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889015-tlp2q_odh-ai-gateway-infra(c3ade289-2392-4add-800c-9070a5a965ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" podUID="c3ade289-2392-4add-800c-9070a5a965ac" Oct 30 06:15:16.853566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:16.853527 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 06:15:16.853779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:16.853591 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 06:15:17.005936 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.005896 5980 scope.go:122] "RemoveContainer" containerID="80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a" Oct 30 06:15:17.006427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.006063 5980 scope.go:122] "RemoveContainer" containerID="972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f" Oct 30 06:15:17.006427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.006241 5980 scope.go:122] "RemoveContainer" containerID="506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1" Oct 30 06:15:17.327728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.327556 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:15:17.598481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.598452 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q"] Oct 30 06:15:17.853775 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.853749 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 06:15:17.853963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.853937 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 06:15:17.907072 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.907048 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:17.953213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.953173 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca\") pod \"c3ade289-2392-4add-800c-9070a5a965ac\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " Oct 30 06:15:17.953420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.953232 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn\" (UniqueName: \"kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn\") pod \"c3ade289-2392-4add-800c-9070a5a965ac\" (UID: \"c3ade289-2392-4add-800c-9070a5a965ac\") " Oct 30 06:15:17.953557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.953533 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca" pod "c3ade289-2392-4add-800c-9070a5a965ac" (UID: "c3ade289-2392-4add-800c-9070a5a965ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:17.953923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.953908 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ade289-2392-4add-800c-9070a5a965ac-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:17.955605 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:17.955574 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn" pod "c3ade289-2392-4add-800c-9070a5a965ac" (UID: "c3ade289-2392-4add-800c-9070a5a965ac"). InnerVolumeSpecName "kube-api-access-vv4nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:18.009430 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.009226 5980 scope.go:122] "RemoveContainer" containerID="9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd" Oct 30 06:15:18.010475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.010342 5980 scope.go:122] "RemoveContainer" containerID="57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2" Oct 30 06:15:18.010572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.010550 5980 scope.go:122] "RemoveContainer" containerID="780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313" Oct 30 06:15:18.056067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.056043 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv4nn\" (UniqueName: \"kubernetes.io/projected/c3ade289-2392-4add-800c-9070a5a965ac-kube-api-access-vv4nn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:18.541637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.541439 5980 generic.go:356] "Generic (PLEG): container finished" podID="fce43331-a061-418e-87ee-cb48df2fc345" containerID="d0768a043b1fb7c80b677e0c905dfad2aa9603b81dfc18fdfd6cd3b3745423e0" exitCode=60 Oct 30 06:15:18.541637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.541507 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" event={"ID":"fce43331-a061-418e-87ee-cb48df2fc345","Type":"ContainerDied","Data":"d0768a043b1fb7c80b677e0c905dfad2aa9603b81dfc18fdfd6cd3b3745423e0"} Oct 30 06:15:18.541637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.541569 5980 scope.go:122] "RemoveContainer" containerID="972a1b9be75566c2006dc67523c859c964bbad09cf3466d9e1ca931081a96e8f" Oct 30 06:15:18.542321 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.542302 5980 scope.go:122] "RemoveContainer" containerID="d0768a043b1fb7c80b677e0c905dfad2aa9603b81dfc18fdfd6cd3b3745423e0" Oct 30 06:15:18.542765 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:18.542738 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w_odh-ai-gateway-infra(fce43331-a061-418e-87ee-cb48df2fc345)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" podUID="fce43331-a061-418e-87ee-cb48df2fc345" Oct 30 06:15:18.543391 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.543377 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" Oct 30 06:15:18.543457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.543419 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q" event={"ID":"c3ade289-2392-4add-800c-9070a5a965ac","Type":"ContainerDied","Data":"3ecf5390889c7f7f779db1bb21a3575c3cfdd7c208c8d99469b3ca8ccf279883"} Oct 30 06:15:18.545659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.545498 5980 generic.go:356] "Generic (PLEG): container finished" podID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerID="2f458c881feddcb1d8538584e3ca3b92144db7e3995cca6d97fad1a4f63c1fb0" exitCode=60 Oct 30 06:15:18.545659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.545600 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" event={"ID":"732c537b-6fa6-40db-aec8-5720bd35f93d","Type":"ContainerDied","Data":"2f458c881feddcb1d8538584e3ca3b92144db7e3995cca6d97fad1a4f63c1fb0"} Oct 30 06:15:18.546023 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.546006 5980 scope.go:122] "RemoveContainer" containerID="2f458c881feddcb1d8538584e3ca3b92144db7e3995cca6d97fad1a4f63c1fb0" Oct 30 06:15:18.546340 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:18.546317 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw_odh-ai-gateway-infra(732c537b-6fa6-40db-aec8-5720bd35f93d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" Oct 30 06:15:18.547775 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.547751 5980 generic.go:356] "Generic (PLEG): container finished" podID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerID="f4e824bcd17d7cd4881778028729d642fe519df5880c3971b3998a6f87b9da22" exitCode=60 Oct 30 06:15:18.547855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.547817 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" event={"ID":"f56bf106-551f-46ec-b4ea-c8f6890dd29b","Type":"ContainerDied","Data":"f4e824bcd17d7cd4881778028729d642fe519df5880c3971b3998a6f87b9da22"} Oct 30 06:15:18.548185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.548171 5980 scope.go:122] "RemoveContainer" containerID="f4e824bcd17d7cd4881778028729d642fe519df5880c3971b3998a6f87b9da22" Oct 30 06:15:18.548419 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:18.548403 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb_odh-ai-gateway-infra(f56bf106-551f-46ec-b4ea-c8f6890dd29b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" Oct 30 06:15:18.554961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.554943 5980 scope.go:122] "RemoveContainer" containerID="1aa5ed037976b0b59ab80cf7926ffa2d822c1d5578fb8afda20a2a93b349b208" Oct 30 06:15:18.593056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.589825 5980 scope.go:122] "RemoveContainer" containerID="506c70d60505afc3051b9310228bddd1091b0bda8db046eee20cd4c1db534fd1" Oct 30 06:15:18.609207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.609146 5980 scope.go:122] "RemoveContainer" containerID="80ae664ae546f582fbef2505e2a05267f71b1cfd715fee5f4bc6f56ffc1a7c2a" Oct 30 06:15:18.711242 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.711217 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q"] Oct 30 06:15:18.714800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.714777 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889015-tlp2q"] Oct 30 06:15:18.853782 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.853752 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 06:15:18.853943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:18.853800 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 06:15:19.006466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.006428 5980 scope.go:122] "RemoveContainer" containerID="b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f" Oct 30 06:15:19.559427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.559347 5980 generic.go:356] "Generic (PLEG): container finished" podID="25609759-f118-43eb-bbd2-3f42e12da38f" containerID="adba684c17c84a6e902d5fac3423438964da0844d8e2ff615ebf16b6547e09c5" exitCode=60 Oct 30 06:15:19.559854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.559419 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerDied","Data":"adba684c17c84a6e902d5fac3423438964da0844d8e2ff615ebf16b6547e09c5"} Oct 30 06:15:19.559854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.559512 5980 scope.go:122] "RemoveContainer" containerID="9f4f2422dd5a7c4ef2d9bddbdb3bce018353ab35b8358d05818e6f220c35cbfd" Oct 30 06:15:19.559928 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.559904 5980 scope.go:122] "RemoveContainer" containerID="adba684c17c84a6e902d5fac3423438964da0844d8e2ff615ebf16b6547e09c5" Oct 30 06:15:19.560295 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:19.560270 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp_odh-ai-gateway-infra(25609759-f118-43eb-bbd2-3f42e12da38f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" Oct 30 06:15:19.561955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.561936 5980 generic.go:356] "Generic (PLEG): container finished" podID="ae284910-6e7d-419b-ad60-814f27522558" containerID="f2ee3e25265b75967dd52f74fa338645c6c7a84e1c141fd13634456339e3b30a" exitCode=60 Oct 30 06:15:19.562073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.562011 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerDied","Data":"f2ee3e25265b75967dd52f74fa338645c6c7a84e1c141fd13634456339e3b30a"} Oct 30 06:15:19.579189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.562446 5980 scope.go:122] "RemoveContainer" containerID="f2ee3e25265b75967dd52f74fa338645c6c7a84e1c141fd13634456339e3b30a" Oct 30 06:15:19.593132 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:19.586381 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk_odh-ai-gateway-infra(ae284910-6e7d-419b-ad60-814f27522558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" podUID="ae284910-6e7d-419b-ad60-814f27522558" Oct 30 06:15:19.597888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.593378 5980 generic.go:356] "Generic (PLEG): container finished" podID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerID="76eb9095bc73c02fdbc0599e43a0439ccb96f074d2cf9e3e553f48bef6bbbdf1" exitCode=60 Oct 30 06:15:19.597888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.593449 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" event={"ID":"38e8d144-5ea8-406c-bed8-c85a44336b68","Type":"ContainerDied","Data":"76eb9095bc73c02fdbc0599e43a0439ccb96f074d2cf9e3e553f48bef6bbbdf1"} Oct 30 06:15:19.597888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.593851 5980 scope.go:122] "RemoveContainer" containerID="76eb9095bc73c02fdbc0599e43a0439ccb96f074d2cf9e3e553f48bef6bbbdf1" Oct 30 06:15:19.597888 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:19.594284 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv_odh-ai-gateway-infra(38e8d144-5ea8-406c-bed8-c85a44336b68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" Oct 30 06:15:19.601668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.601638 5980 scope.go:122] "RemoveContainer" containerID="780740134e4d11b7a488ca4233b6cc06ba6f33928ef7f4f7467578b9590d8313" Oct 30 06:15:19.617335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.617306 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w"] Oct 30 06:15:19.617826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.617799 5980 scope.go:122] "RemoveContainer" containerID="57e896e25d97759cde5d639f58c00f65d2f0881f7ea6405a26bd2803248e52c2" Oct 30 06:15:19.709680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.708058 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb"] Oct 30 06:15:19.813709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.813621 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw"] Oct 30 06:15:19.862150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.861800 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 06:15:19.862150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.861852 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 06:15:19.933041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.933011 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv"] Oct 30 06:15:19.934267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.934246 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.934398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.934388 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.934473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.934465 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.934536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.934526 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.934663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.934651 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.934752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.934741 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.935023 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.935008 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.935145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.935132 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:19.939578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.938995 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:19.954139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.954052 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:20.005132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.978137 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv"] Oct 30 06:15:20.005528 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:19.982815 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c73fcb-openshift-default-dockercfg-rtknk\"" Oct 30 06:15:20.020268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.020230 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv\" (UniqueName: \"kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv\") pod \"fce43331-a061-418e-87ee-cb48df2fc345\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " Oct 30 06:15:20.020868 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.020836 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca\") pod \"fce43331-a061-418e-87ee-cb48df2fc345\" (UID: \"fce43331-a061-418e-87ee-cb48df2fc345\") " Oct 30 06:15:20.024723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.024696 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.025079 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.025059 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.025351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.025325 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.025529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.025514 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.025717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.025703 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.026164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.026143 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.027086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.021451 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca" pod "fce43331-a061-418e-87ee-cb48df2fc345" (UID: "fce43331-a061-418e-87ee-cb48df2fc345"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.027086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.020441 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ade289-2392-4add-800c-9070a5a965ac" path="/var/lib/kubelet/pods/c3ade289-2392-4add-800c-9070a5a965ac/volumes" Oct 30 06:15:20.027727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.027351 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.027727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.027407 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.027727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.027430 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4bx\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.027727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.027571 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce43331-a061-418e-87ee-cb48df2fc345-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.028718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.028695 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv" pod "fce43331-a061-418e-87ee-cb48df2fc345" (UID: "fce43331-a061-418e-87ee-cb48df2fc345"). InnerVolumeSpecName "kube-api-access-x2cqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.134554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.134523 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.134839 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.134818 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.135021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.135004 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.135194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.135174 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.135543 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.135434 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.135927 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.135734 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.135438 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136087 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136137 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4bx\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136194 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136280 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136432 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136615 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2cqv\" (UniqueName: \"kubernetes.io/projected/fce43331-a061-418e-87ee-cb48df2fc345-kube-api-access-x2cqv\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.137310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.136879 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.138640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.138383 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.138904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.138862 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:20.141353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.141335 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.141608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.141594 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.161600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.161475 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.161978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.161956 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4bx\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.233011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.232976 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:20.237410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.237378 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq\" (UniqueName: \"kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq\") pod \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " Oct 30 06:15:20.237545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.237489 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca\") pod \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\" (UID: \"f56bf106-551f-46ec-b4ea-c8f6890dd29b\") " Oct 30 06:15:20.239041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.238960 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca" pod "f56bf106-551f-46ec-b4ea-c8f6890dd29b" (UID: "f56bf106-551f-46ec-b4ea-c8f6890dd29b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.240683 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.240659 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq" pod "f56bf106-551f-46ec-b4ea-c8f6890dd29b" (UID: "f56bf106-551f-46ec-b4ea-c8f6890dd29b"). InnerVolumeSpecName "kube-api-access-s2knq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.313673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.313602 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:20.339886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.339859 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca\") pod \"732c537b-6fa6-40db-aec8-5720bd35f93d\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " Oct 30 06:15:20.340018 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.339914 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj\" (UniqueName: \"kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj\") pod \"732c537b-6fa6-40db-aec8-5720bd35f93d\" (UID: \"732c537b-6fa6-40db-aec8-5720bd35f93d\") " Oct 30 06:15:20.340316 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.340288 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca" pod "732c537b-6fa6-40db-aec8-5720bd35f93d" (UID: "732c537b-6fa6-40db-aec8-5720bd35f93d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.340706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.340687 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2knq\" (UniqueName: \"kubernetes.io/projected/f56bf106-551f-46ec-b4ea-c8f6890dd29b-kube-api-access-s2knq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.340790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.340712 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c537b-6fa6-40db-aec8-5720bd35f93d-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.340790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.340726 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f56bf106-551f-46ec-b4ea-c8f6890dd29b-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.342705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.342677 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj" pod "732c537b-6fa6-40db-aec8-5720bd35f93d" (UID: "732c537b-6fa6-40db-aec8-5720bd35f93d"). InnerVolumeSpecName "kube-api-access-pcsqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.442500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.442468 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcsqj\" (UniqueName: \"kubernetes.io/projected/732c537b-6fa6-40db-aec8-5720bd35f93d-kube-api-access-pcsqj\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.613166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.612666 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" event={"ID":"732c537b-6fa6-40db-aec8-5720bd35f93d","Type":"ContainerDied","Data":"20ab3f9c7605316b1e5a2b962a141732dd1419197df504f54b24c706e59fd816"} Oct 30 06:15:20.613768 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.613743 5980 scope.go:122] "RemoveContainer" containerID="2f458c881feddcb1d8538584e3ca3b92144db7e3995cca6d97fad1a4f63c1fb0" Oct 30 06:15:20.613953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.613754 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw" Oct 30 06:15:20.616648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.616615 5980 generic.go:356] "Generic (PLEG): container finished" podID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerID="aceb2b1fa19770a1adb48c7595bcdee68fc7d1b00bf5c0a5667a59879f04f16e" exitCode=60 Oct 30 06:15:20.616755 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.616715 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" event={"ID":"a733495c-f173-4a82-b3a2-cb6bf48aa83d","Type":"ContainerDied","Data":"aceb2b1fa19770a1adb48c7595bcdee68fc7d1b00bf5c0a5667a59879f04f16e"} Oct 30 06:15:20.617301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.617282 5980 scope.go:122] "RemoveContainer" containerID="aceb2b1fa19770a1adb48c7595bcdee68fc7d1b00bf5c0a5667a59879f04f16e" Oct 30 06:15:20.617684 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:20.617658 5980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4_odh-ai-gateway-infra(a733495c-f173-4a82-b3a2-cb6bf48aa83d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" Oct 30 06:15:20.632100 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.632049 5980 scope.go:122] "RemoveContainer" containerID="b9eeef02de7568c602c4e432e9e7419d9ff958edc567f7e78e6d5555cc72ca9f" Oct 30 06:15:20.635207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.634692 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp"] Oct 30 06:15:20.642314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.642154 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" event={"ID":"f56bf106-551f-46ec-b4ea-c8f6890dd29b","Type":"ContainerDied","Data":"b1b6f810304d078095271363098ab79d3153ca6933bb088faaafb28bcc96ec15"} Oct 30 06:15:20.659097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.658946 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb" Oct 30 06:15:20.662248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.661629 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" event={"ID":"fce43331-a061-418e-87ee-cb48df2fc345","Type":"ContainerDied","Data":"4b2d104c0ca6bc3fe41e291ca403a7bd4fbec43d9345bd52cedb4e17df8a09d1"} Oct 30 06:15:20.662248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.661720 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w" Oct 30 06:15:20.685782 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.685409 5980 scope.go:122] "RemoveContainer" containerID="f4e824bcd17d7cd4881778028729d642fe519df5880c3971b3998a6f87b9da22" Oct 30 06:15:20.691970 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:15:20.686616 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403e3648_157f_404c_b8d2_42204133c0a7.slice/crio-b47ddd405167e9306486ef4a4008735caebf49924fcceb8f41d14e6f78e105c4 WatchSource:0}: Error finding container b47ddd405167e9306486ef4a4008735caebf49924fcceb8f41d14e6f78e105c4: Status 404 returned error can't find the container with id b47ddd405167e9306486ef4a4008735caebf49924fcceb8f41d14e6f78e105c4 Oct 30 06:15:20.691970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.689245 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:20.691970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.689365 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:20.691970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.689410 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:20.691970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.689975 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv"] Oct 30 06:15:20.697031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.695408 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv"] Oct 30 06:15:20.720588 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.720356 5980 scope.go:122] "RemoveContainer" containerID="d0768a043b1fb7c80b677e0c905dfad2aa9603b81dfc18fdfd6cd3b3745423e0" Oct 30 06:15:20.749524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.749490 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk"] Oct 30 06:15:20.842967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.842937 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w"] Oct 30 06:15:20.867181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.866540 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-25d881-29889015-9zl2w"] Oct 30 06:15:20.876175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.876149 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb"] Oct 30 06:15:20.927578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:20.911275 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889015-pnpxb"] Oct 30 06:15:21.020295 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.018259 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw"] Oct 30 06:15:21.028512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.028477 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f22a52-29889015-l4gsw"] Oct 30 06:15:21.073187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.069441 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:21.073187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.071076 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:21.084754 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.073666 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:15:21.095996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.095918 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:21.179184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.174152 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj\" (UniqueName: \"kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj\") pod \"25609759-f118-43eb-bbd2-3f42e12da38f\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " Oct 30 06:15:21.179184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.174326 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca\") pod \"25609759-f118-43eb-bbd2-3f42e12da38f\" (UID: \"25609759-f118-43eb-bbd2-3f42e12da38f\") " Oct 30 06:15:21.179184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.176910 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca" pod "25609759-f118-43eb-bbd2-3f42e12da38f" (UID: "25609759-f118-43eb-bbd2-3f42e12da38f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.228706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.228663 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj" pod "25609759-f118-43eb-bbd2-3f42e12da38f" (UID: "25609759-f118-43eb-bbd2-3f42e12da38f"). InnerVolumeSpecName "kube-api-access-jtkpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.246867 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.246840 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:21.253434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.253410 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:21.277698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.277605 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkpj\" (UniqueName: \"kubernetes.io/projected/25609759-f118-43eb-bbd2-3f42e12da38f-kube-api-access-jtkpj\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.277958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.277936 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25609759-f118-43eb-bbd2-3f42e12da38f-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.383283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.383243 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67\" (UniqueName: \"kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67\") pod \"ae284910-6e7d-419b-ad60-814f27522558\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " Oct 30 06:15:21.387779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.387399 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn\" (UniqueName: \"kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn\") pod \"38e8d144-5ea8-406c-bed8-c85a44336b68\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " Oct 30 06:15:21.401053 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.400960 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca" pod "ae284910-6e7d-419b-ad60-814f27522558" (UID: "ae284910-6e7d-419b-ad60-814f27522558"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.401327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.401306 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca\") pod \"ae284910-6e7d-419b-ad60-814f27522558\" (UID: \"ae284910-6e7d-419b-ad60-814f27522558\") " Oct 30 06:15:21.402378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.402352 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca\") pod \"38e8d144-5ea8-406c-bed8-c85a44336b68\" (UID: \"38e8d144-5ea8-406c-bed8-c85a44336b68\") " Oct 30 06:15:21.406046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.402804 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca" pod "38e8d144-5ea8-406c-bed8-c85a44336b68" (UID: "38e8d144-5ea8-406c-bed8-c85a44336b68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.406587 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.406347 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67" pod "ae284910-6e7d-419b-ad60-814f27522558" (UID: "ae284910-6e7d-419b-ad60-814f27522558"). InnerVolumeSpecName "kube-api-access-gdm67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.411548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.411503 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae284910-6e7d-419b-ad60-814f27522558-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.411701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.411687 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e8d144-5ea8-406c-bed8-c85a44336b68-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.411802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.411791 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdm67\" (UniqueName: \"kubernetes.io/projected/ae284910-6e7d-419b-ad60-814f27522558-kube-api-access-gdm67\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.415516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.415495 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn" pod "38e8d144-5ea8-406c-bed8-c85a44336b68" (UID: "38e8d144-5ea8-406c-bed8-c85a44336b68"). InnerVolumeSpecName "kube-api-access-c9mtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.514477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.514390 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mtn\" (UniqueName: \"kubernetes.io/projected/38e8d144-5ea8-406c-bed8-c85a44336b68-kube-api-access-c9mtn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.691874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.691836 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" event={"ID":"403e3648-157f-404c-b8d2-42204133c0a7","Type":"ContainerStarted","Data":"7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662"} Oct 30 06:15:21.692461 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.692189 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" event={"ID":"403e3648-157f-404c-b8d2-42204133c0a7","Type":"ContainerStarted","Data":"b47ddd405167e9306486ef4a4008735caebf49924fcceb8f41d14e6f78e105c4"} Oct 30 06:15:21.693369 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.693347 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" Oct 30 06:15:21.693502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.693353 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp" event={"ID":"25609759-f118-43eb-bbd2-3f42e12da38f","Type":"ContainerDied","Data":"3ceae96a1dd56e2dd9464beea27ebbabf9a5236ef7c89ccb06023e74c81d778a"} Oct 30 06:15:21.693563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.693504 5980 scope.go:122] "RemoveContainer" containerID="adba684c17c84a6e902d5fac3423438964da0844d8e2ff615ebf16b6547e09c5" Oct 30 06:15:21.694903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.694692 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" Oct 30 06:15:21.694903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.694713 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk" event={"ID":"ae284910-6e7d-419b-ad60-814f27522558","Type":"ContainerDied","Data":"54db3d3722ddd8d11bc12c5d4f05dd8d17cc714c23d9ad500467fe0bac2dbc33"} Oct 30 06:15:21.697443 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.697255 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" Oct 30 06:15:21.697682 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.697267 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv" event={"ID":"38e8d144-5ea8-406c-bed8-c85a44336b68","Type":"ContainerDied","Data":"79a0a8061f8d378de540a086598388ab583643f9b82853148b22305d4fa0fe14"} Oct 30 06:15:21.708269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.708249 5980 scope.go:122] "RemoveContainer" containerID="f2ee3e25265b75967dd52f74fa338645c6c7a84e1c141fd13634456339e3b30a" Oct 30 06:15:21.709468 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.709325 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4"] Oct 30 06:15:21.718169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.718148 5980 scope.go:122] "RemoveContainer" containerID="76eb9095bc73c02fdbc0599e43a0439ccb96f074d2cf9e3e553f48bef6bbbdf1" Oct 30 06:15:21.754083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.754033 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podStartSLOduration=2.754018986 podStartE2EDuration="2.754018986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:21.752977583 +0000 UTC m=+2575.861016643" watchObservedRunningTime="2026-10-30 06:15:21.754018986 +0000 UTC m=+2575.862058019" Oct 30 06:15:21.813411 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.813292 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv"] Oct 30 06:15:21.819696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.819582 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e2e3d0-29889015-4kxvv"] Oct 30 06:15:21.854512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.854479 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk"] Oct 30 06:15:21.861225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.861199 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889015-cpqvk"] Oct 30 06:15:21.938856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.938831 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:21.958982 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.958956 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp"] Oct 30 06:15:21.978271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:21.976555 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889015-t8gkp"] Oct 30 06:15:22.016929 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.016890 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" path="/var/lib/kubelet/pods/25609759-f118-43eb-bbd2-3f42e12da38f/volumes" Oct 30 06:15:22.017689 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.017661 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" path="/var/lib/kubelet/pods/38e8d144-5ea8-406c-bed8-c85a44336b68/volumes" Oct 30 06:15:22.018234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.018215 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" path="/var/lib/kubelet/pods/732c537b-6fa6-40db-aec8-5720bd35f93d/volumes" Oct 30 06:15:22.019656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.019227 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae284910-6e7d-419b-ad60-814f27522558" path="/var/lib/kubelet/pods/ae284910-6e7d-419b-ad60-814f27522558/volumes" Oct 30 06:15:22.019830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.019806 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" path="/var/lib/kubelet/pods/f56bf106-551f-46ec-b4ea-c8f6890dd29b/volumes" Oct 30 06:15:22.020407 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.020384 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce43331-a061-418e-87ee-cb48df2fc345" path="/var/lib/kubelet/pods/fce43331-a061-418e-87ee-cb48df2fc345/volumes" Oct 30 06:15:22.026649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.026621 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn\" (UniqueName: \"kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn\") pod \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " Oct 30 06:15:22.026765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.026672 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca\") pod \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\" (UID: \"a733495c-f173-4a82-b3a2-cb6bf48aa83d\") " Oct 30 06:15:22.027146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.027097 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca" pod "a733495c-f173-4a82-b3a2-cb6bf48aa83d" (UID: "a733495c-f173-4a82-b3a2-cb6bf48aa83d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:22.027578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.027556 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a733495c-f173-4a82-b3a2-cb6bf48aa83d-maas-api-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.029946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.029919 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn" pod "a733495c-f173-4a82-b3a2-cb6bf48aa83d" (UID: "a733495c-f173-4a82-b3a2-cb6bf48aa83d"). InnerVolumeSpecName "kube-api-access-c7vgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:22.131724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.131660 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vgn\" (UniqueName: \"kubernetes.io/projected/a733495c-f173-4a82-b3a2-cb6bf48aa83d-kube-api-access-c7vgn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.313857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.313832 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:22.315960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.315930 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:15:22.316096 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.315987 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 06:15:22.710955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.710911 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" Oct 30 06:15:22.711509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.710921 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4" event={"ID":"a733495c-f173-4a82-b3a2-cb6bf48aa83d","Type":"ContainerDied","Data":"874af00facb0116a0a999d10d80185c952d84b0190d28b86152c3c6e847076b4"} Oct 30 06:15:22.711509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.711057 5980 scope.go:122] "RemoveContainer" containerID="aceb2b1fa19770a1adb48c7595bcdee68fc7d1b00bf5c0a5667a59879f04f16e" Oct 30 06:15:22.790134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.789156 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4"] Oct 30 06:15:22.796221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:22.796185 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0f980e-29889015-gwdc4"] Oct 30 06:15:23.319135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:23.317437 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:15:23.319135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:23.317499 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 06:15:24.016439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:24.016401 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" path="/var/lib/kubelet/pods/a733495c-f173-4a82-b3a2-cb6bf48aa83d/volumes" Oct 30 06:15:24.315336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:24.314888 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:15:24.315336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:24.314939 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 06:15:25.314405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:25.314367 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:15:25.314837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:25.314445 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 06:15:26.314706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:26.314676 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:15:26.315135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:26.314728 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 06:15:27.314858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:27.314828 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 30 06:15:27.315400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:27.314896 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 30 06:15:28.319618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:28.319591 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:28.320061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:28.320039 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:28.320905 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:28.320886 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:15:39.818576 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.818541 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx"] Oct 30 06:15:39.819269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819253 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819270 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819291 5980 cpu_manager.go:414] "containerMap: removing container" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819299 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819307 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819312 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819324 5980 cpu_manager.go:414] "containerMap: removing container" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819330 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819339 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819345 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819351 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819355 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819363 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819367 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819380 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819385 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819393 5980 cpu_manager.go:414] "containerMap: removing container" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819399 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819407 5980 cpu_manager.go:414] "containerMap: removing container" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819411 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819420 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819424 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819431 5980 cpu_manager.go:414] "containerMap: removing container" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819435 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819444 5980 cpu_manager.go:414] "containerMap: removing container" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819448 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819458 5980 cpu_manager.go:414] "containerMap: removing container" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819462 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819470 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819474 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819484 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819488 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819497 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819500 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819601 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819610 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819615 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819621 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819625 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819631 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819636 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819643 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819647 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819653 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ade289-2392-4add-800c-9070a5a965ac" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819659 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819665 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819670 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819676 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a733495c-f173-4a82-b3a2-cb6bf48aa83d" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819682 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819689 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:15:39.819809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.819698 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:15:39.825210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.825194 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:39.854466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.854448 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-dfc3d2-serving-cert\"" Oct 30 06:15:39.885393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.885360 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx"] Oct 30 06:15:39.993953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.993916 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9xz\" (UniqueName: \"kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:39.994140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.993987 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:39.994202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.994177 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:39.994354 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:39.994338 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.096571 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.096537 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.096896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.096871 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9xz\" (UniqueName: \"kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.097086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.096936 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.097086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.097046 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.098064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.098033 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.112981 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.099530 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.118164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.114839 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.124133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.123506 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9xz\" (UniqueName: \"kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz\") pod \"maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.139417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.139393 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:40.468711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.468648 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx"] Oct 30 06:15:40.469038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.469014 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:15:40.933467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:40.933437 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" event={"ID":"4896357c-db54-4af4-9c1c-d08559326b79","Type":"ContainerStarted","Data":"b721b8ce649d63817d0862442a26e34de0690dff411a7e9181fa5cdad0b0ad4f"} Oct 30 06:15:41.941965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:41.941924 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" event={"ID":"4896357c-db54-4af4-9c1c-d08559326b79","Type":"ContainerStarted","Data":"f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7"} Oct 30 06:15:41.942369 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:41.942020 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:41.991900 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:41.991850 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" podStartSLOduration=2.585498389 podStartE2EDuration="2.991838973s" totalImagesPullingTime="406.340584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:41.99154392 +0000 UTC m=+2596.099582953" watchObservedRunningTime="2026-10-30 06:15:41.991838973 +0000 UTC m=+2596.099878005" Oct 30 06:15:43.092190 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:15:43.092155 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:15:47.956522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:47.956487 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:15:55.395297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:55.395252 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:15:55.395854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:55.395342 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:15:57.186587 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:57.186554 5980 generic.go:356] "Generic (PLEG): container finished" podID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerID="f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d" exitCode=0 Oct 30 06:15:57.187050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:57.186627 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" event={"ID":"efb5598b-7e14-4635-be7a-8d80130a25ae","Type":"ContainerDied","Data":"f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d"} Oct 30 06:15:57.187050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:57.187033 5980 scope.go:122] "RemoveContainer" containerID="f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d" Oct 30 06:15:57.188764 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:57.188734 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:57.188826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:57.188811 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:57.188871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:57.188844 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:58.199433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:58.199393 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" event={"ID":"efb5598b-7e14-4635-be7a-8d80130a25ae","Type":"ContainerStarted","Data":"2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1"} Oct 30 06:15:58.721656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:58.721617 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:15:58.721876 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:58.721857 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 06:15:58.721959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:58.721915 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 06:15:59.721391 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:59.721361 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 06:15:59.721795 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:15:59.721422 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 06:16:00.721976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:00.721943 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 06:16:00.722402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:00.722005 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 06:16:01.721371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:01.721334 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 06:16:01.721553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:01.721393 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 06:16:02.721698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:02.721665 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 06:16:02.722208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:02.721719 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 06:16:03.204918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:03.204880 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl"] Oct 30 06:16:03.780367 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:03.780336 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:16:03.781577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:03.781553 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:16:03.783614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:03.783591 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:16:06.181340 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.181191 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t"] Oct 30 06:16:06.183468 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.183442 5980 cpu_manager.go:414] "containerMap: removing container" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:16:06.183651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.183637 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:16:06.183789 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.183777 5980 cpu_manager.go:414] "containerMap: removing container" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:16:06.183898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.183886 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:16:06.184065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.184043 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:16:06.184177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.184163 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce43331-a061-418e-87ee-cb48df2fc345" containerName="cleanup" Oct 30 06:16:06.184328 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.184317 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:16:06.184422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.184412 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:16:06.184716 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.184702 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae284910-6e7d-419b-ad60-814f27522558" containerName="cleanup" Oct 30 06:16:06.184847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.184836 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e8d144-5ea8-406c-bed8-c85a44336b68" containerName="cleanup" Oct 30 06:16:06.188251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.188231 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c537b-6fa6-40db-aec8-5720bd35f93d" containerName="cleanup" Oct 30 06:16:06.188418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.188406 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="25609759-f118-43eb-bbd2-3f42e12da38f" containerName="cleanup" Oct 30 06:16:06.188553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.188542 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56bf106-551f-46ec-b4ea-c8f6890dd29b" containerName="cleanup" Oct 30 06:16:06.212427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.212399 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.258484 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.222536 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t"] Oct 30 06:16:06.426647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.397860 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f7ad21-openshift-default-dockercfg-7qtgl\"" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433235 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433335 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433377 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433451 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzb96\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433544 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433781 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433947 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.433980 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.437565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.434054 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535330 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535532 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535406 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535532 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535435 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535532 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535491 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzb96\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535532 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535518 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535590 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535700 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535728 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535757 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535765 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.535915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.535809 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.536124 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.536092 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.536261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.536141 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.536437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.536361 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.538024 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.538001 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.538264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.538247 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.560724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.560701 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.561436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.561378 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzb96\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96\") pod \"e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:06.739972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:06.739879 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:07.145547 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:16:07.145515 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa79cc6_da0a_481f_af58_8777a4ab1e7b.slice/crio-97b1ef9df759f50a3db2927463891ad859cef7f2f4049ea237e629f11eeb3076 WatchSource:0}: Error finding container 97b1ef9df759f50a3db2927463891ad859cef7f2f4049ea237e629f11eeb3076: Status 404 returned error can't find the container with id 97b1ef9df759f50a3db2927463891ad859cef7f2f4049ea237e629f11eeb3076 Oct 30 06:16:07.146138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.146089 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t"] Oct 30 06:16:07.148908 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.148863 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:07.149002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.148945 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:07.149002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.148983 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:07.445782 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.445673 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" event={"ID":"1fa79cc6-da0a-481f-af58-8777a4ab1e7b","Type":"ContainerStarted","Data":"a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3"} Oct 30 06:16:07.446312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.445793 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" event={"ID":"1fa79cc6-da0a-481f-af58-8777a4ab1e7b","Type":"ContainerStarted","Data":"97b1ef9df759f50a3db2927463891ad859cef7f2f4049ea237e629f11eeb3076"} Oct 30 06:16:07.496394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.496335 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podStartSLOduration=1.496315815 podStartE2EDuration="1.496315815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:07.494771497 +0000 UTC m=+2621.602810556" watchObservedRunningTime="2026-10-30 06:16:07.496315815 +0000 UTC m=+2621.604354850" Oct 30 06:16:07.740679 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:07.740602 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:08.206062 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.206020 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" podUID="8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" containerName="payload-pre-processing" containerID="cri-o://a31ebd3860144506d23ac7af24195efcadd4c56fdcdf029009d529a674447372" gracePeriod=25 Oct 30 06:16:08.456281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.456194 5980 generic.go:356] "Generic (PLEG): container finished" podID="8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" containerID="a31ebd3860144506d23ac7af24195efcadd4c56fdcdf029009d529a674447372" exitCode=0 Oct 30 06:16:08.456657 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.456266 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" event={"ID":"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da","Type":"ContainerDied","Data":"a31ebd3860144506d23ac7af24195efcadd4c56fdcdf029009d529a674447372"} Oct 30 06:16:08.500574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.500546 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:16:08.575651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.575617 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm\" (UniqueName: \"kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm\") pod \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " Oct 30 06:16:08.575825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.575688 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume\") pod \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\" (UID: \"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da\") " Oct 30 06:16:08.576094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.576067 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume" pod "8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" (UID: "8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:08.576825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.576796 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.599172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.591360 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm" pod "8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" (UID: "8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da"). InnerVolumeSpecName "kube-api-access-lqsfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:08.680280 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.679136 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqsfm\" (UniqueName: \"kubernetes.io/projected/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da-kube-api-access-lqsfm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.747286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:08.747206 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:09.466437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.466391 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" event={"ID":"8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da","Type":"ContainerDied","Data":"0f2e074f1482573abdd47d82a07da6cb640a299bbe5ec2a29fa9d16168462ef9"} Oct 30 06:16:09.466952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.466459 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl" Oct 30 06:16:09.466952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.466474 5980 scope.go:122] "RemoveContainer" containerID="a31ebd3860144506d23ac7af24195efcadd4c56fdcdf029009d529a674447372" Oct 30 06:16:09.466952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.466483 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:09.467731 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.467707 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:16:09.586153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.579215 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl"] Oct 30 06:16:09.586153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:09.585469 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-5b6f8cb7b7-j59kl"] Oct 30 06:16:10.016186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:10.016145 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" path="/var/lib/kubelet/pods/8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da/volumes" Oct 30 06:16:10.658040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:10.658005 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2"] Oct 30 06:16:10.658531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:10.658220 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" containerID="cri-o://5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca" gracePeriod=30 Oct 30 06:16:13.090843 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:16:13.090805 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:16:14.218335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:14.218293 5980 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:16:14.218728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:14.218387 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:16:15.985527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:15.985501 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 06:16:16.169369 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169333 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169482 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169515 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169555 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169679 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169724 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169808 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169863 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.169989 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.169914 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy\") pod \"d4f878bc-d34c-450b-a038-1956f16d58ad\" (UID: \"d4f878bc-d34c-450b-a038-1956f16d58ad\") " Oct 30 06:16:16.170284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.170256 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:16.170676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.170622 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.171032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.171003 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:16.171202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.171181 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:16.171426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.171402 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:16.171426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.171412 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:16.174323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.173314 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:16.174323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.173967 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "kube-api-access-5gxrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:16.174323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.174167 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:16:16.181968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.181942 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy" pod "d4f878bc-d34c-450b-a038-1956f16d58ad" (UID: "d4f878bc-d34c-450b-a038-1956f16d58ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272226 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4f878bc-d34c-450b-a038-1956f16d58ad-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272257 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272269 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272285 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272295 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272306 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272317 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxrd\" (UniqueName: \"kubernetes.io/projected/d4f878bc-d34c-450b-a038-1956f16d58ad-kube-api-access-5gxrd\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.272669 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.272330 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4f878bc-d34c-450b-a038-1956f16d58ad-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:16.595260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.591683 5980 generic.go:356] "Generic (PLEG): container finished" podID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerID="5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca" exitCode=0 Oct 30 06:16:16.595260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.591816 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" event={"ID":"d4f878bc-d34c-450b-a038-1956f16d58ad","Type":"ContainerDied","Data":"5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca"} Oct 30 06:16:16.595260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.591867 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" event={"ID":"d4f878bc-d34c-450b-a038-1956f16d58ad","Type":"ContainerDied","Data":"293739758e433aca593faab6c8c595f20401b4ed2cfe7077233de6d57deb4e90"} Oct 30 06:16:16.595260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.591897 5980 scope.go:122] "RemoveContainer" containerID="5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca" Oct 30 06:16:16.595260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.592043 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2" Oct 30 06:16:16.748140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.745293 5980 scope.go:122] "RemoveContainer" containerID="5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca" Oct 30 06:16:16.749235 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:16:16.749202 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca\": container with ID starting with 5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca not found: ID does not exist" containerID="5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca" Oct 30 06:16:16.749434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.749406 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca"} err="failed to get container status \"5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca\": rpc error: code = NotFound desc = could not find container \"5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca\": container with ID starting with 5c6487fed3d6687c9bc987761d39a9707c5c653414e71dfc84d35202776677ca not found: ID does not exist" Oct 30 06:16:16.767239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.767201 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2"] Oct 30 06:16:16.786139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:16.783488 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-gv4h2"] Oct 30 06:16:16.886145 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:16:16.886034 5980 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f878bc_d34c_450b_a038_1956f16d58ad.slice/crio-293739758e433aca593faab6c8c595f20401b4ed2cfe7077233de6d57deb4e90\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f878bc_d34c_450b_a038_1956f16d58ad.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:16:18.016510 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:18.016474 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" path="/var/lib/kubelet/pods/d4f878bc-d34c-450b-a038-1956f16d58ad/volumes" Oct 30 06:16:24.657614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:24.657571 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2"] Oct 30 06:16:27.972897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:27.972861 5980 scope.go:122] "RemoveContainer" containerID="c7afd97fa2d3abac641ca69114f303a38772c77aac95d415eca8d319176c57a6" Oct 30 06:16:29.658570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:29.658502 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" podUID="3cb45ba3-169c-4e79-ba7b-e3400c307aae" containerName="payload-pre-processing" containerID="cri-o://c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a" gracePeriod=25 Oct 30 06:16:30.002870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.002843 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:16:30.141345 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.141297 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp\" (UniqueName: \"kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp\") pod \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " Oct 30 06:16:30.146470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.141425 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume\") pod \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\" (UID: \"3cb45ba3-169c-4e79-ba7b-e3400c307aae\") " Oct 30 06:16:30.146470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.141765 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume" pod "3cb45ba3-169c-4e79-ba7b-e3400c307aae" (UID: "3cb45ba3-169c-4e79-ba7b-e3400c307aae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:30.146470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.142493 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb45ba3-169c-4e79-ba7b-e3400c307aae-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:30.146470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.143726 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp" pod "3cb45ba3-169c-4e79-ba7b-e3400c307aae" (UID: "3cb45ba3-169c-4e79-ba7b-e3400c307aae"). InnerVolumeSpecName "kube-api-access-g9jtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:30.244099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.244010 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jtp\" (UniqueName: \"kubernetes.io/projected/3cb45ba3-169c-4e79-ba7b-e3400c307aae-kube-api-access-g9jtp\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:16:30.761552 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.761515 5980 generic.go:356] "Generic (PLEG): container finished" podID="3cb45ba3-169c-4e79-ba7b-e3400c307aae" containerID="c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a" exitCode=0 Oct 30 06:16:30.762014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.761573 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" Oct 30 06:16:30.762014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.761597 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" event={"ID":"3cb45ba3-169c-4e79-ba7b-e3400c307aae","Type":"ContainerDied","Data":"c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a"} Oct 30 06:16:30.762014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.761670 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2" event={"ID":"3cb45ba3-169c-4e79-ba7b-e3400c307aae","Type":"ContainerDied","Data":"a3cf20673bd1d48b25a4587529ec1875d603854e50179bc611033fd8ab957e01"} Oct 30 06:16:30.762014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.761693 5980 scope.go:122] "RemoveContainer" containerID="c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a" Oct 30 06:16:30.763823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.763802 5980 generic.go:356] "Generic (PLEG): container finished" podID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerID="8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e" exitCode=0 Oct 30 06:16:30.763943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.763880 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" event={"ID":"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0","Type":"ContainerDied","Data":"8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e"} Oct 30 06:16:30.764328 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.764311 5980 scope.go:122] "RemoveContainer" containerID="8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e" Oct 30 06:16:30.766432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.766396 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:30.766532 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.766490 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:30.766594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.766538 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:30.774540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.774520 5980 scope.go:122] "RemoveContainer" containerID="c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a" Oct 30 06:16:30.774824 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:16:30.774794 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a\": container with ID starting with c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a not found: ID does not exist" containerID="c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a" Oct 30 06:16:30.775005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.774833 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a"} err="failed to get container status \"c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a\": rpc error: code = NotFound desc = could not find container \"c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a\": container with ID starting with c3016019d0f7e3d6d1063451535455e815bf639977731cbfe248f278f86cd31a not found: ID does not exist" Oct 30 06:16:30.823335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.823272 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2"] Oct 30 06:16:30.830064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:30.830040 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-5b65b8fdf5-cwtw2"] Oct 30 06:16:31.821575 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:31.821537 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" event={"ID":"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0","Type":"ContainerStarted","Data":"1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f"} Oct 30 06:16:32.015163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:32.015134 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb45ba3-169c-4e79-ba7b-e3400c307aae" path="/var/lib/kubelet/pods/3cb45ba3-169c-4e79-ba7b-e3400c307aae/volumes" Oct 30 06:16:32.144311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:32.144228 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:16:32.161418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:32.161389 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:16:32.175422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:32.175370 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:16:32.175762 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:32.175729 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:16:33.162135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:33.161367 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:16:33.162135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:33.161424 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:16:34.161866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:34.161834 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.162042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:34.161877 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:16:35.161537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:35.161509 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:16:35.162054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:35.162030 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:16:36.162132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:36.162074 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:16:36.162625 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:36.162179 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:16:37.170136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:37.165743 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 06:16:37.170136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:37.165816 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 06:16:38.165933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:38.165899 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:16:38.166911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:38.166885 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:16:43.141507 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:16:43.141440 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:16:43.910591 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.910556 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c"] Oct 30 06:16:43.911296 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911282 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" Oct 30 06:16:43.911358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911298 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" Oct 30 06:16:43.911358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911320 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3cb45ba3-169c-4e79-ba7b-e3400c307aae" containerName="payload-pre-processing" Oct 30 06:16:43.911358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911325 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb45ba3-169c-4e79-ba7b-e3400c307aae" containerName="payload-pre-processing" Oct 30 06:16:43.911358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911340 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" containerName="payload-pre-processing" Oct 30 06:16:43.911358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911345 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" containerName="payload-pre-processing" Oct 30 06:16:43.911524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911464 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4f878bc-d34c-450b-a038-1956f16d58ad" containerName="istio-proxy" Oct 30 06:16:43.911524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911476 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb45ba3-169c-4e79-ba7b-e3400c307aae" containerName="payload-pre-processing" Oct 30 06:16:43.911524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.911484 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9f14a6-2a69-43f8-a8fc-ef9ec6ecb3da" containerName="payload-pre-processing" Oct 30 06:16:43.914621 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.914599 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:43.936944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.936915 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c"] Oct 30 06:16:43.944564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.944539 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"default-dockercfg-xgkjl\"" Oct 30 06:16:43.944732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.944643 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"openshift-service-ca.crt\"" Oct 30 06:16:43.944732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.944714 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"kube-root-ca.crt\"" Oct 30 06:16:43.945247 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.945227 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"rate-test-model-d1bfce-kserve-self-signed-certs\"" Oct 30 06:16:43.988951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.988919 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:43.989083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.989070 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6hlc\" (UniqueName: \"kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:43.989182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.989166 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:43.989321 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.989308 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:43.989374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.989363 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:43.989411 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:43.989386 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090453 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090524 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090546 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090579 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090700 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6hlc\" (UniqueName: \"kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090787 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.090968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090948 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.091022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.090974 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.091324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.091302 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.092955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.092925 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.094065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.094040 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.116383 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.116359 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6hlc\" (UniqueName: \"kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc\") pod \"rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.230187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.229398 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:44.622838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.622811 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c"] Oct 30 06:16:44.623273 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:16:44.623249 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc9fc10_b4da_4639_90da_9c6661ea2dbc.slice/crio-5b5524e95bea722b38a059068d038dc94b1651f1a91253a2e306bb3fde5c7f6a WatchSource:0}: Error finding container 5b5524e95bea722b38a059068d038dc94b1651f1a91253a2e306bb3fde5c7f6a: Status 404 returned error can't find the container with id 5b5524e95bea722b38a059068d038dc94b1651f1a91253a2e306bb3fde5c7f6a Oct 30 06:16:44.982239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.982091 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" event={"ID":"8fc9fc10-b4da-4639-90da-9c6661ea2dbc","Type":"ContainerStarted","Data":"28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472"} Oct 30 06:16:44.982239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:44.982196 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" event={"ID":"8fc9fc10-b4da-4639-90da-9c6661ea2dbc","Type":"ContainerStarted","Data":"5b5524e95bea722b38a059068d038dc94b1651f1a91253a2e306bb3fde5c7f6a"} Oct 30 06:16:45.030734 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:45.030682 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" podStartSLOduration=2.030667356 podStartE2EDuration="2.030667356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:45.028160342 +0000 UTC m=+2659.136199375" watchObservedRunningTime="2026-10-30 06:16:45.030667356 +0000 UTC m=+2659.138706635" Oct 30 06:16:45.230425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:45.230398 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:54.229818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:54.229788 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:54.242488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:54.242461 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:16:55.071607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:16:55.071578 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:17:02.538775 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.538740 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks"] Oct 30 06:17:02.543272 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.543253 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.567632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.567604 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks"] Oct 30 06:17:02.577194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.576354 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f7ad21-serving-cert\"" Oct 30 06:17:02.597426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.597392 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.597589 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.597520 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.597661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.597632 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tmx\" (UniqueName: \"kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.597840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.597819 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.699800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.699764 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.700029 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.700010 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.700203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.700183 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.700306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.700231 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tmx\" (UniqueName: \"kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.700417 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:17:02.700377 5980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-serving-cert: secret "maas-api-e2e-ipp-b-f7ad21-serving-cert" not found Oct 30 06:17:02.700489 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:17:02.700470 5980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls podName:76c9f39c-2e68-41a5-a7ab-95d3bb40e653 nodeName:}" failed. No retries permitted until 2026-10-30 06:17:03.200438118 +0000 UTC m=+2677.308477144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls") pod "maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" (UID: "76c9f39c-2e68-41a5-a7ab-95d3bb40e653") : secret "maas-api-e2e-ipp-b-f7ad21-serving-cert" not found Oct 30 06:17:02.700761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.700738 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.702664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.702642 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:02.725118 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:02.725079 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tmx\" (UniqueName: \"kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:03.209565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:03.209529 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:03.212155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:03.212080 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:03.460070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:03.459986 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:03.804872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:03.804844 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks"] Oct 30 06:17:03.805472 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:03.805445 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c9f39c_2e68_41a5_a7ab_95d3bb40e653.slice/crio-42faa0d7f794f5b0eb550bd4d596d2a5f7c2c95ac332bd66a45184f9f90d239f WatchSource:0}: Error finding container 42faa0d7f794f5b0eb550bd4d596d2a5f7c2c95ac332bd66a45184f9f90d239f: Status 404 returned error can't find the container with id 42faa0d7f794f5b0eb550bd4d596d2a5f7c2c95ac332bd66a45184f9f90d239f Oct 30 06:17:04.152822 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:04.152781 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" event={"ID":"76c9f39c-2e68-41a5-a7ab-95d3bb40e653","Type":"ContainerStarted","Data":"42faa0d7f794f5b0eb550bd4d596d2a5f7c2c95ac332bd66a45184f9f90d239f"} Oct 30 06:17:05.161404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:05.161368 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" event={"ID":"76c9f39c-2e68-41a5-a7ab-95d3bb40e653","Type":"ContainerStarted","Data":"7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd"} Oct 30 06:17:05.161838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:05.161464 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:05.210929 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:05.210872 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" podStartSLOduration=2.797401807 podStartE2EDuration="3.210854587s" totalImagesPullingTime="413.45278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:05.209831454 +0000 UTC m=+2679.317870486" watchObservedRunningTime="2026-10-30 06:17:05.210854587 +0000 UTC m=+2679.318893623" Oct 30 06:17:11.183566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:11.183535 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:17:12.586251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:12.586213 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv"] Oct 30 06:17:13.094983 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:17:13.094947 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:17:27.858029 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:27.857981 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" podUID="20858ec4-dbca-43b1-adab-4e6d3fb4aacc" containerName="main" containerID="cri-o://faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65" gracePeriod=45 Oct 30 06:17:28.171570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.171532 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:17:28.301846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.301816 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs\" (UniqueName: \"kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs\") pod \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " Oct 30 06:17:28.302050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.301868 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home\") pod \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " Oct 30 06:17:28.302050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.301895 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir\") pod \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " Oct 30 06:17:28.302050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.301938 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2\") pod \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " Oct 30 06:17:28.302050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.301987 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm\") pod \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " Oct 30 06:17:28.302620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.302079 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache\") pod \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\" (UID: \"20858ec4-dbca-43b1-adab-4e6d3fb4aacc\") " Oct 30 06:17:28.302620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.302594 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache" pod "20858ec4-dbca-43b1-adab-4e6d3fb4aacc" (UID: "20858ec4-dbca-43b1-adab-4e6d3fb4aacc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:28.302852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.302814 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir" pod "20858ec4-dbca-43b1-adab-4e6d3fb4aacc" (UID: "20858ec4-dbca-43b1-adab-4e6d3fb4aacc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:28.303083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.303063 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:17:28.303371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.303091 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:17:28.303371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.303177 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home" pod "20858ec4-dbca-43b1-adab-4e6d3fb4aacc" (UID: "20858ec4-dbca-43b1-adab-4e6d3fb4aacc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:28.305394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.305371 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs" pod "20858ec4-dbca-43b1-adab-4e6d3fb4aacc" (UID: "20858ec4-dbca-43b1-adab-4e6d3fb4aacc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:28.305668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.305648 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2" pod "20858ec4-dbca-43b1-adab-4e6d3fb4aacc" (UID: "20858ec4-dbca-43b1-adab-4e6d3fb4aacc"). InnerVolumeSpecName "kube-api-access-smnb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:28.305747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.305647 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm" pod "20858ec4-dbca-43b1-adab-4e6d3fb4aacc" (UID: "20858ec4-dbca-43b1-adab-4e6d3fb4aacc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:28.405099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.405031 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:17:28.405099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.405056 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:17:28.405099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.405066 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnb2\" (UniqueName: \"kubernetes.io/projected/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-kube-api-access-smnb2\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:17:28.405099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.405075 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20858ec4-dbca-43b1-adab-4e6d3fb4aacc-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:17:28.593186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.583806 5980 generic.go:356] "Generic (PLEG): container finished" podID="20858ec4-dbca-43b1-adab-4e6d3fb4aacc" containerID="faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65" exitCode=0 Oct 30 06:17:28.593186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.584052 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" event={"ID":"20858ec4-dbca-43b1-adab-4e6d3fb4aacc","Type":"ContainerDied","Data":"faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65"} Oct 30 06:17:28.593186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.584140 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" event={"ID":"20858ec4-dbca-43b1-adab-4e6d3fb4aacc","Type":"ContainerDied","Data":"8be0d2e1375b15519f9f18f9d43fffe7d6e191fccc8f70bfac3ea89cbcfb8a39"} Oct 30 06:17:28.593186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.584169 5980 scope.go:122] "RemoveContainer" containerID="faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65" Oct 30 06:17:28.593186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.584399 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv" Oct 30 06:17:28.622433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.622409 5980 scope.go:122] "RemoveContainer" containerID="faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65" Oct 30 06:17:28.624037 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:17:28.624006 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65\": container with ID starting with faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65 not found: ID does not exist" containerID="faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65" Oct 30 06:17:28.624206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.624049 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65"} err="failed to get container status \"faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65\": rpc error: code = NotFound desc = could not find container \"faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65\": container with ID starting with faca4be2dd7fa0c1c46e5b3719e6df90ec13f40dd30d3b07a396ec45951f5f65 not found: ID does not exist" Oct 30 06:17:28.658182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.658099 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv"] Oct 30 06:17:28.661802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:28.661783 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-0f980e/auth-test-model-0f980e-kserve-7686f99945-rlrbv"] Oct 30 06:17:30.016091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:30.016055 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20858ec4-dbca-43b1-adab-4e6d3fb4aacc" path="/var/lib/kubelet/pods/20858ec4-dbca-43b1-adab-4e6d3fb4aacc/volumes" Oct 30 06:17:33.153308 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.153277 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf"] Oct 30 06:17:33.154061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.154041 5980 cpu_manager.go:414] "containerMap: removing container" podUID="20858ec4-dbca-43b1-adab-4e6d3fb4aacc" containerName="main" Oct 30 06:17:33.154061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.154059 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20858ec4-dbca-43b1-adab-4e6d3fb4aacc" containerName="main" Oct 30 06:17:33.154286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.154251 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="20858ec4-dbca-43b1-adab-4e6d3fb4aacc" containerName="main" Oct 30 06:17:33.159334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.159308 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.183951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.183928 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d51cd8-openshift-default-dockercfg-gdp6w\"" Oct 30 06:17:33.196664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.195963 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf"] Oct 30 06:17:33.248301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248260 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8pq8\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248323 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248382 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248602 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248495 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248602 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248559 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248605 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248679 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248737 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.248831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.248804 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.365384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.350063 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.386960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.386926 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.387273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.387249 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.387473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.387457 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8pq8\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.387626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.387610 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.387796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.387781 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.388070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.388052 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.388374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.388278 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.388374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.388348 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.395284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.395209 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.403770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.403031 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.403973 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.403892 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.404505 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.404486 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.409991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.409967 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.439367 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.412562 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.439699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.438409 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.444314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.444289 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.471128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.445066 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8pq8\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.476420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.476391 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:33.947400 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:33.947363 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47573111_9bc0_43e8_b490_f6d908e928fe.slice/crio-322bfebf979ced9a5968af1ad1e9f47d7272d5416e1ae14f1a1ee89817f9d62b WatchSource:0}: Error finding container 322bfebf979ced9a5968af1ad1e9f47d7272d5416e1ae14f1a1ee89817f9d62b: Status 404 returned error can't find the container with id 322bfebf979ced9a5968af1ad1e9f47d7272d5416e1ae14f1a1ee89817f9d62b Oct 30 06:17:33.947606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.947585 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf"] Oct 30 06:17:33.950696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.950660 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:33.950784 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.950743 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:33.950827 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:33.950783 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:34.646413 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:34.646369 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" event={"ID":"47573111-9bc0-43e8-b490-f6d908e928fe","Type":"ContainerStarted","Data":"ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0"} Oct 30 06:17:34.646860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:34.646442 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" event={"ID":"47573111-9bc0-43e8-b490-f6d908e928fe","Type":"ContainerStarted","Data":"322bfebf979ced9a5968af1ad1e9f47d7272d5416e1ae14f1a1ee89817f9d62b"} Oct 30 06:17:34.724280 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:34.724236 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podStartSLOduration=1.7242207920000001 podStartE2EDuration="1.724220792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:34.709606825 +0000 UTC m=+2708.817645852" watchObservedRunningTime="2026-10-30 06:17:34.724220792 +0000 UTC m=+2708.832259825" Oct 30 06:17:35.485288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:35.485257 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:35.492574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:35.492548 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:35.665881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:35.665800 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:35.667848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:35.667820 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:17:39.165763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.165723 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2"] Oct 30 06:17:39.170562 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.170540 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.195152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.195081 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2"] Oct 30 06:17:39.201121 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.201077 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d51cd8-dockercfg-f9k5q\"" Oct 30 06:17:39.201739 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.201715 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d51cd8\"" Oct 30 06:17:39.211371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.211350 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4"] Oct 30 06:17:39.216461 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.216312 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.244586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.244549 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4"] Oct 30 06:17:39.380141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.380026 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prv28\" (UniqueName: \"kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.380141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.380075 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.380393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.380297 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.380393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.380378 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwxp\" (UniqueName: \"kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp\") pod \"payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.484183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.483389 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prv28\" (UniqueName: \"kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.484183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.483450 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.484183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.483644 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.484183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.483714 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwxp\" (UniqueName: \"kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp\") pod \"payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.499131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.485167 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.499131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.485851 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.565198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.554224 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwxp\" (UniqueName: \"kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp\") pod \"payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.574149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.572386 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prv28\" (UniqueName: \"kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:39.789908 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.789813 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:39.843774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:39.843747 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:40.102507 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:40.102481 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08df514_f3c2_4c6e_9b36_b45e36d9440d.slice/crio-659929635b90fb66ad1bac1e00dd3478192a4daf0d0d9dfdce714715ead04239 WatchSource:0}: Error finding container 659929635b90fb66ad1bac1e00dd3478192a4daf0d0d9dfdce714715ead04239: Status 404 returned error can't find the container with id 659929635b90fb66ad1bac1e00dd3478192a4daf0d0d9dfdce714715ead04239 Oct 30 06:17:40.107013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.106986 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2"] Oct 30 06:17:40.183252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.183226 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4"] Oct 30 06:17:40.184674 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:40.184089 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05169d2_d0b4_4ee8_b746_ccc3f2df157f.slice/crio-11192538e524d1deaf8c208e82aab694c03c1b26809788c94fbfc009cda95341 WatchSource:0}: Error finding container 11192538e524d1deaf8c208e82aab694c03c1b26809788c94fbfc009cda95341: Status 404 returned error can't find the container with id 11192538e524d1deaf8c208e82aab694c03c1b26809788c94fbfc009cda95341 Oct 30 06:17:40.713723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.713688 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" event={"ID":"b05169d2-d0b4-4ee8-b746-ccc3f2df157f","Type":"ContainerStarted","Data":"13321c9f757e5f5870eb67b1dc0a3150be4944b5bbbb3d4ce928f0a7d28108f5"} Oct 30 06:17:40.713955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.713752 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:40.713955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.713773 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" event={"ID":"b05169d2-d0b4-4ee8-b746-ccc3f2df157f","Type":"ContainerStarted","Data":"11192538e524d1deaf8c208e82aab694c03c1b26809788c94fbfc009cda95341"} Oct 30 06:17:40.715471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.715442 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" event={"ID":"f08df514-f3c2-4c6e-9b36-b45e36d9440d","Type":"ContainerStarted","Data":"04e09edd0a4336905d686ee003d5d41794e3999ec61106fb8d842779c07d21b0"} Oct 30 06:17:40.715584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.715490 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:40.715584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.715506 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" event={"ID":"f08df514-f3c2-4c6e-9b36-b45e36d9440d","Type":"ContainerStarted","Data":"659929635b90fb66ad1bac1e00dd3478192a4daf0d0d9dfdce714715ead04239"} Oct 30 06:17:40.767429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.767371 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" podStartSLOduration=1.767355866 podStartE2EDuration="1.767355866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:40.763910439 +0000 UTC m=+2714.871949473" watchObservedRunningTime="2026-10-30 06:17:40.767355866 +0000 UTC m=+2714.875394899" Oct 30 06:17:40.809301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:40.809248 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" podStartSLOduration=1.809233425 podStartE2EDuration="1.809233425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:40.80863477 +0000 UTC m=+2714.916673804" watchObservedRunningTime="2026-10-30 06:17:40.809233425 +0000 UTC m=+2714.917272472" Oct 30 06:17:43.090606 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:17:43.090573 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:17:43.628548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.628516 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs"] Oct 30 06:17:43.652480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.652446 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.665165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.657960 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs"] Oct 30 06:17:43.683295 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.683271 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"openshift-service-ca.crt\"" Oct 30 06:17:43.683445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.683327 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"rate-test-model-24db22-kserve-self-signed-certs\"" Oct 30 06:17:43.683445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.683377 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"default-dockercfg-ngbhd\"" Oct 30 06:17:43.683445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.683414 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"kube-root-ca.crt\"" Oct 30 06:17:43.757502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.757467 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.757677 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.757658 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.757773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.757756 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.757852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.757841 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.757888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.757874 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd46n\" (UniqueName: \"kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.757933 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.757891 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859151 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859241 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859289 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859324 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd46n\" (UniqueName: \"kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859342 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859548 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859531 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859752 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859811 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.859928 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.859883 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.861579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.861555 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.861766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.861748 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.899763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.899685 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd46n\" (UniqueName: \"kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n\") pod \"rate-test-model-24db22-kserve-7b8c9944d9-gtgvs\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:43.984465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:43.984429 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:44.393896 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:44.393860 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5052f3d9_b903_410f_83ba_7b0cd797e710.slice/crio-9ae360727d9cf62b8a31f25abeae925c8ff1564e9a99cff8fca273419d98453f WatchSource:0}: Error finding container 9ae360727d9cf62b8a31f25abeae925c8ff1564e9a99cff8fca273419d98453f: Status 404 returned error can't find the container with id 9ae360727d9cf62b8a31f25abeae925c8ff1564e9a99cff8fca273419d98453f Oct 30 06:17:44.394333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:44.394068 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs"] Oct 30 06:17:44.745377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:44.745280 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" event={"ID":"5052f3d9-b903-410f-83ba-7b0cd797e710","Type":"ContainerStarted","Data":"a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d"} Oct 30 06:17:44.745377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:44.745323 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" event={"ID":"5052f3d9-b903-410f-83ba-7b0cd797e710","Type":"ContainerStarted","Data":"9ae360727d9cf62b8a31f25abeae925c8ff1564e9a99cff8fca273419d98453f"} Oct 30 06:17:44.794359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:44.794300 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" podStartSLOduration=1.794283109 podStartE2EDuration="1.794283109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:44.790726815 +0000 UTC m=+2718.898765847" watchObservedRunningTime="2026-10-30 06:17:44.794283109 +0000 UTC m=+2718.902322142" Oct 30 06:17:44.985715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:44.985681 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:45.112813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.112784 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw"] Oct 30 06:17:45.117851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.117826 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.135798 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.135777 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5deb36-openshift-default-dockercfg-7vlwt\"" Oct 30 06:17:45.151400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.151370 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw"] Oct 30 06:17:45.192775 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.192737 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.192984 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.192868 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.192984 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.192936 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.193148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.193058 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.193221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.193199 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.193318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.193292 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.193529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.193512 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.193620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.193583 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp77c\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.193683 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.193617 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304228 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp77c\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304288 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304362 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304442 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304510 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304566 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304638 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304725 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.308213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.304880 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.323132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.312419 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.323132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.316451 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.323132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.316704 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.323132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.316949 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.323132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.317206 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.364181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.346433 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.364181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.363772 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp77c\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.371142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.369667 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.390130 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.374405 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.441131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.440406 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:45.933471 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:45.933439 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377612ce_6b65_4347_949f_d03e31a4c596.slice/crio-7b21974cee7b0d9d4671380844e6341125ec72aa50de0ebfb5c787a79f551e59 WatchSource:0}: Error finding container 7b21974cee7b0d9d4671380844e6341125ec72aa50de0ebfb5c787a79f551e59: Status 404 returned error can't find the container with id 7b21974cee7b0d9d4671380844e6341125ec72aa50de0ebfb5c787a79f551e59 Oct 30 06:17:45.933615 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.933526 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw"] Oct 30 06:17:45.936861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.936827 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:45.936951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.936896 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:45.936951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:45.936929 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:46.761935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:46.761893 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" event={"ID":"377612ce-6b65-4347-949f-d03e31a4c596","Type":"ContainerStarted","Data":"30107178ffb80b3edd9b8257918d65a04c5835ca07e68ca109bcd14ed6bed061"} Oct 30 06:17:46.762456 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:46.761945 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" event={"ID":"377612ce-6b65-4347-949f-d03e31a4c596","Type":"ContainerStarted","Data":"7b21974cee7b0d9d4671380844e6341125ec72aa50de0ebfb5c787a79f551e59"} Oct 30 06:17:46.826270 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:46.826222 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podStartSLOduration=1.826208083 podStartE2EDuration="1.826208083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:46.823161696 +0000 UTC m=+2720.931200729" watchObservedRunningTime="2026-10-30 06:17:46.826208083 +0000 UTC m=+2720.934247175" Oct 30 06:17:47.302671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.302636 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr"] Oct 30 06:17:47.308844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.308807 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.326492 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.326459 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"openshift-service-ca.crt\"" Oct 30 06:17:47.326655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.326587 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"kube-root-ca.crt\"" Oct 30 06:17:47.332035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.332009 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr"] Oct 30 06:17:47.337871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.337851 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"rate-test-model-91f42c-kserve-self-signed-certs\"" Oct 30 06:17:47.337988 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.337890 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"default-dockercfg-hgwk4\"" Oct 30 06:17:47.365599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.365564 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.365769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.365747 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv4v2\" (UniqueName: \"kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.365916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.365896 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.366089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.366064 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.366306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.366286 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.366419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.366400 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.441829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.441786 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:47.448664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.448638 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:47.467585 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.467558 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.467757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.467627 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.467757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.467726 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv4v2\" (UniqueName: \"kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.467874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.467843 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.467988 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.467957 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.468400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.468378 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.468779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.468661 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.468969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.468950 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.469042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.468100 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.471369 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.471343 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.473460 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.473430 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.498031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.498000 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv4v2\" (UniqueName: \"kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2\") pod \"rate-test-model-91f42c-kserve-6db4f58f69-pvgmr\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.640188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.640053 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:47.772678 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.772647 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:47.778193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:47.776980 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:17:48.087740 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:48.087711 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a2a0442_d5e7_4823_9ad3_11c53b921113.slice/crio-890a3d3411dc764d3812e76ac894be26c15e8c3ef5eac1a81ee034352f272c0a WatchSource:0}: Error finding container 890a3d3411dc764d3812e76ac894be26c15e8c3ef5eac1a81ee034352f272c0a: Status 404 returned error can't find the container with id 890a3d3411dc764d3812e76ac894be26c15e8c3ef5eac1a81ee034352f272c0a Oct 30 06:17:48.089886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:48.089859 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr"] Oct 30 06:17:48.785316 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:48.785281 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" event={"ID":"2a2a0442-d5e7-4823-9ad3-11c53b921113","Type":"ContainerStarted","Data":"f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df"} Oct 30 06:17:48.785316 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:48.785320 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" event={"ID":"2a2a0442-d5e7-4823-9ad3-11c53b921113","Type":"ContainerStarted","Data":"890a3d3411dc764d3812e76ac894be26c15e8c3ef5eac1a81ee034352f272c0a"} Oct 30 06:17:48.831069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:48.831010 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" podStartSLOduration=1.830992448 podStartE2EDuration="1.830992448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:48.829069098 +0000 UTC m=+2722.937108130" watchObservedRunningTime="2026-10-30 06:17:48.830992448 +0000 UTC m=+2722.939031483" Oct 30 06:17:49.642134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:49.641185 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:51.728886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:51.728839 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:17:51.731184 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:51.731161 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:17:54.017144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:53.996632 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:54.053815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:54.053782 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:54.896728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:54.896695 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:17:55.809959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.808341 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq"] Oct 30 06:17:55.821394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.813313 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:55.836191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.835539 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq"] Oct 30 06:17:55.842682 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.842656 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f6302c\"/\"openshift-service-ca.crt\"" Oct 30 06:17:55.843033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.843009 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-f6302c\"/\"kube-root-ca.crt\"" Oct 30 06:17:55.844186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.844170 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f6302c\"/\"default-dockercfg-nh4pv\"" Oct 30 06:17:55.844186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.844170 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-f6302c\"/\"auth-test-model-f6302c-kserve-self-signed-certs\"" Oct 30 06:17:55.938417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.938378 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:55.938627 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.938608 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:55.938701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.938643 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:55.938782 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.938755 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj74t\" (UniqueName: \"kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:55.938886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.938867 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:55.939331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:55.939042 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.040770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.040733 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.041250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.041219 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.041425 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.041403 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.041856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.041838 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.041956 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.041794 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.041956 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.041909 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj74t\" (UniqueName: \"kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.042188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.041964 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.042188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.042168 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.044373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.042942 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.125294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.121967 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.148130 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.147843 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj74t\" (UniqueName: \"kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.192132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.173313 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm\") pod \"auth-test-model-f6302c-kserve-844cb99849-kghfq\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.436340 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.436269 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:56.795586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.795560 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq"] Oct 30 06:17:56.798271 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:17:56.798230 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ed193f_7ac3_4428_9051_59a8e1b62b4e.slice/crio-4c1ca1192bf75e4cd270f610835f518d60b96dcbc02e301a16dc62684192bce7 WatchSource:0}: Error finding container 4c1ca1192bf75e4cd270f610835f518d60b96dcbc02e301a16dc62684192bce7: Status 404 returned error can't find the container with id 4c1ca1192bf75e4cd270f610835f518d60b96dcbc02e301a16dc62684192bce7 Oct 30 06:17:56.917177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.916969 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" event={"ID":"39ed193f-7ac3-4428-9051-59a8e1b62b4e","Type":"ContainerStarted","Data":"bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a"} Oct 30 06:17:56.917177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.917019 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" event={"ID":"39ed193f-7ac3-4428-9051-59a8e1b62b4e","Type":"ContainerStarted","Data":"4c1ca1192bf75e4cd270f610835f518d60b96dcbc02e301a16dc62684192bce7"} Oct 30 06:17:56.996180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:56.996096 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" podStartSLOduration=1.996075018 podStartE2EDuration="1.996075018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:56.982883955 +0000 UTC m=+2731.090922988" watchObservedRunningTime="2026-10-30 06:17:56.996075018 +0000 UTC m=+2731.104114052" Oct 30 06:17:57.436746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:57.436707 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:17:57.640324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:57.640295 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:57.654792 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:57.654762 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:57.936428 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:57.936394 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:17:58.467823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:58.467787 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8"] Oct 30 06:17:58.468325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:17:58.468293 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" containerID="cri-o://69943a8901e79c65754be8503c0b81ebc042b2c203bcfcf6703bd0a506a3bf78" gracePeriod=30 Oct 30 06:18:04.082133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.058179 5980 generic.go:356] "Generic (PLEG): container finished" podID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerID="69943a8901e79c65754be8503c0b81ebc042b2c203bcfcf6703bd0a506a3bf78" exitCode=0 Oct 30 06:18:04.162053 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.162012 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" event={"ID":"004a7fd2-38f9-4522-b994-73b7596e11c7","Type":"ContainerDied","Data":"69943a8901e79c65754be8503c0b81ebc042b2c203bcfcf6703bd0a506a3bf78"} Oct 30 06:18:04.405172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.405142 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:18:04.474523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474489 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474716 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474576 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474716 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474613 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474716 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474704 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474816 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474905 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474873 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474922 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.474999 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.474956 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.475173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.475130 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk\") pod \"004a7fd2-38f9-4522-b994-73b7596e11c7\" (UID: \"004a7fd2-38f9-4522-b994-73b7596e11c7\") " Oct 30 06:18:04.475751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.475668 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:04.477009 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.476887 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.479305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.478286 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:04.479305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.478469 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:04.479305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.479275 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:04.479842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.479810 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:04.484547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.484519 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:18:04.487553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.487526 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:04.487785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.487767 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:04.487870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.487856 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk" pod "004a7fd2-38f9-4522-b994-73b7596e11c7" (UID: "004a7fd2-38f9-4522-b994-73b7596e11c7"). InnerVolumeSpecName "kube-api-access-sqxbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:04.591899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591845 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.591899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591884 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.591899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591898 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/004a7fd2-38f9-4522-b994-73b7596e11c7-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.592699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591912 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.592699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591924 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.592699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591936 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.592699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591948 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/004a7fd2-38f9-4522-b994-73b7596e11c7-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:04.592699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:04.591963 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxbk\" (UniqueName: \"kubernetes.io/projected/004a7fd2-38f9-4522-b994-73b7596e11c7-kube-api-access-sqxbk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:05.099939 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:05.099902 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" event={"ID":"004a7fd2-38f9-4522-b994-73b7596e11c7","Type":"ContainerDied","Data":"7316d04248a1c6dbc7908a1358588ad7e624d8c4043bd42e3aacc308651f3ab5"} Oct 30 06:18:05.111414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:05.100522 5980 scope.go:122] "RemoveContainer" containerID="69943a8901e79c65754be8503c0b81ebc042b2c203bcfcf6703bd0a506a3bf78" Oct 30 06:18:05.111414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:05.100718 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8" Oct 30 06:18:05.169846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:05.169803 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8"] Oct 30 06:18:05.174597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:05.174575 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-x7cg8"] Oct 30 06:18:06.017709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.017680 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" path="/var/lib/kubelet/pods/004a7fd2-38f9-4522-b994-73b7596e11c7/volumes" Oct 30 06:18:06.110125 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.110081 5980 generic.go:356] "Generic (PLEG): container finished" podID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerID="a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3" exitCode=0 Oct 30 06:18:06.110521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.110160 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" event={"ID":"1fa79cc6-da0a-481f-af58-8777a4ab1e7b","Type":"ContainerDied","Data":"a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3"} Oct 30 06:18:06.110569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.110553 5980 scope.go:122] "RemoveContainer" containerID="a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3" Oct 30 06:18:06.112364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.112334 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:06.112432 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.112408 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:06.112472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.112456 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:06.437058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.437025 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:18:06.450360 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:06.450335 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:18:07.137773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:07.137729 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" event={"ID":"1fa79cc6-da0a-481f-af58-8777a4ab1e7b","Type":"ContainerStarted","Data":"b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6"} Oct 30 06:18:07.152493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:07.152461 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:18:07.740086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:07.740053 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:18:07.741279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:07.741254 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 06:18:07.741465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:07.741307 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 06:18:08.740670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:08.740641 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 06:18:08.741122 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:08.740697 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 06:18:09.499235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:09.499200 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:18:09.741052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:09.741017 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 06:18:09.741603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:09.741076 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 06:18:10.740872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:10.740835 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 06:18:10.741060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:10.740907 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 06:18:11.740287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:11.740236 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 06:18:11.740802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:11.740305 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 06:18:12.740960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:12.740927 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 06:18:12.741381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:12.740981 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 06:18:13.101291 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:18:13.101245 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:18:13.746457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:13.746429 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:18:13.747793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:13.747772 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:18:16.932869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:16.932830 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8"] Oct 30 06:18:16.933355 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:16.933174 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" containerID="cri-o://07afd0ac45644d744e82d61bd210edfe849452bcaa09095d7361f0b5c4dd30c1" gracePeriod=30 Oct 30 06:18:19.439199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:19.439164 5980 patch_prober.go:29] interesting pod/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:18:19.439596 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:19.439220 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:18:22.383433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.383355 5980 generic.go:356] "Generic (PLEG): container finished" podID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerID="07afd0ac45644d744e82d61bd210edfe849452bcaa09095d7361f0b5c4dd30c1" exitCode=0 Oct 30 06:18:22.383433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.383428 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" event={"ID":"fc060397-36c4-4861-9c97-19eb24b3da3a","Type":"ContainerDied","Data":"07afd0ac45644d744e82d61bd210edfe849452bcaa09095d7361f0b5c4dd30c1"} Oct 30 06:18:22.383964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.383510 5980 scope.go:122] "RemoveContainer" containerID="01839a1aafb985273da099aaded6e2a66344322117565f3b458ca34d280d9669" Oct 30 06:18:22.484817 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.484787 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:18:22.587264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.587168 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.587662 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.587643 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.587940 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.587925 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.588089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.588075 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.588224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.588209 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.588390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.588368 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.588542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.588515 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.588676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.588663 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.588909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.588887 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq\") pod \"fc060397-36c4-4861-9c97-19eb24b3da3a\" (UID: \"fc060397-36c4-4861-9c97-19eb24b3da3a\") " Oct 30 06:18:22.594536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.594335 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:22.594738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.594691 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:22.594974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.594946 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:22.595342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.595321 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:22.595506 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.595489 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:22.603645 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.603606 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "kube-api-access-92sjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:22.620166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.611916 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:18:22.762064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.762035 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.762344 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.762327 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.762472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.762459 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.762580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.762569 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.762693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.762682 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.777197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.777169 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc060397-36c4-4861-9c97-19eb24b3da3a-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.777427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.777412 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92sjq\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-kube-api-access-92sjq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.804272 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.804238 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:22.823394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.823339 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy" pod "fc060397-36c4-4861-9c97-19eb24b3da3a" (UID: "fc060397-36c4-4861-9c97-19eb24b3da3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:22.926191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.915524 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:22.926191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:22.915557 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc060397-36c4-4861-9c97-19eb24b3da3a-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:23.398131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:23.397966 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" event={"ID":"fc060397-36c4-4861-9c97-19eb24b3da3a","Type":"ContainerDied","Data":"52ca5d112c998c67d9bb672e5958c9c4a278f2b76c038805f5fdee94cd928ea7"} Oct 30 06:18:23.398131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:23.398037 5980 scope.go:122] "RemoveContainer" containerID="07afd0ac45644d744e82d61bd210edfe849452bcaa09095d7361f0b5c4dd30c1" Oct 30 06:18:23.398685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:23.398255 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8" Oct 30 06:18:23.512805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:23.512759 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8"] Oct 30 06:18:23.533229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:23.533194 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-db0540-openshift-default-5d778c79bf-d2nv8"] Oct 30 06:18:24.017412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:24.017376 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" path="/var/lib/kubelet/pods/fc060397-36c4-4861-9c97-19eb24b3da3a/volumes" Oct 30 06:18:43.092561 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:18:43.092521 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:18:50.638507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:50.638474 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw"] Oct 30 06:18:50.639314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:50.639262 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" containerID="cri-o://2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1" gracePeriod=30 Oct 30 06:18:55.990804 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:55.990772 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.138665 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.138723 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.138882 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.138920 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.138964 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139310 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139360 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139411 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.140998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139468 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket\") pod \"efb5598b-7e14-4635-be7a-8d80130a25ae\" (UID: \"efb5598b-7e14-4635-be7a-8d80130a25ae\") " Oct 30 06:18:56.142470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139313 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:56.142470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139466 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:56.142667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.139617 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:56.142667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.141417 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:56.142667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.141966 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:56.144028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.144003 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:56.144392 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.144364 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "kube-api-access-8r2rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:56.144490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.144400 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:56.146086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.146060 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo" pod "efb5598b-7e14-4635-be7a-8d80130a25ae" (UID: "efb5598b-7e14-4635-be7a-8d80130a25ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242145 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242169 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242177 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb5598b-7e14-4635-be7a-8d80130a25ae-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242186 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242195 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242203 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242211 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242218 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb5598b-7e14-4635-be7a-8d80130a25ae-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.242500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.242226 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2rg\" (UniqueName: \"kubernetes.io/projected/efb5598b-7e14-4635-be7a-8d80130a25ae-kube-api-access-8r2rg\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:18:56.743979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.743944 5980 generic.go:356] "Generic (PLEG): container finished" podID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerID="2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1" exitCode=0 Oct 30 06:18:56.744201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.743993 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" Oct 30 06:18:56.744201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.744033 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" event={"ID":"efb5598b-7e14-4635-be7a-8d80130a25ae","Type":"ContainerDied","Data":"2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1"} Oct 30 06:18:56.744201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.744091 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw" event={"ID":"efb5598b-7e14-4635-be7a-8d80130a25ae","Type":"ContainerDied","Data":"f4a56ce82c93a209c265380dd190cecd050483fbd40c27f73836e20303bac911"} Oct 30 06:18:56.744201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.744166 5980 scope.go:122] "RemoveContainer" containerID="2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1" Oct 30 06:18:56.756035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.756018 5980 scope.go:122] "RemoveContainer" containerID="f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d" Oct 30 06:18:56.765900 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.765780 5980 scope.go:122] "RemoveContainer" containerID="2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1" Oct 30 06:18:56.766096 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:18:56.766070 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1\": container with ID starting with 2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1 not found: ID does not exist" containerID="2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1" Oct 30 06:18:56.766169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.766140 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1"} err="failed to get container status \"2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1\": rpc error: code = NotFound desc = could not find container \"2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1\": container with ID starting with 2c6e8d42800e17d3985b12d7158e0d6e534fe131788a4c7a94290c47de1eeec1 not found: ID does not exist" Oct 30 06:18:56.766169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.766160 5980 scope.go:122] "RemoveContainer" containerID="f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d" Oct 30 06:18:56.766416 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:18:56.766394 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d\": container with ID starting with f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d not found: ID does not exist" containerID="f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d" Oct 30 06:18:56.766475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.766426 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d"} err="failed to get container status \"f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d\": rpc error: code = NotFound desc = could not find container \"f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d\": container with ID starting with f284ced246c9446154ce1fd22ea391ec69ca44dc0f70ace292b9fa1c9ee04b9d not found: ID does not exist" Oct 30 06:18:56.806825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.806803 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw"] Oct 30 06:18:56.812828 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:56.812797 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2d4576-openshift-default-779b666c6f-5nxvw"] Oct 30 06:18:58.015484 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:18:58.015453 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" path="/var/lib/kubelet/pods/efb5598b-7e14-4635-be7a-8d80130a25ae/volumes" Oct 30 06:19:01.683086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:01.683051 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm"] Oct 30 06:19:01.717793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:01.717756 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj"] Oct 30 06:19:06.686768 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:06.686674 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerName="payload-processing" containerID="cri-o://5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24" gracePeriod=25 Oct 30 06:19:06.718233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:06.718165 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" podUID="ba6d5408-690c-4535-9740-64d45c107c05" containerName="payload-pre-processing" containerID="cri-o://12e6a8948ffebe796189004c3f041f0b94f48f0dea34a53c6fdb4b001d8fc948" gracePeriod=25 Oct 30 06:19:10.638174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:10.638133 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:19:11.646384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:11.646345 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" podUID="ba6d5408-690c-4535-9740-64d45c107c05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:19:13.091040 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:19:13.090998 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:19:14.172073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172039 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d"] Oct 30 06:19:14.172883 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172863 5980 cpu_manager.go:414] "containerMap: removing container" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172885 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172900 5980 cpu_manager.go:414] "containerMap: removing container" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172905 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172923 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172927 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172941 5980 cpu_manager.go:414] "containerMap: removing container" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172946 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172983 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" Oct 30 06:19:14.173037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.172988 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" Oct 30 06:19:14.173428 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.173137 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="004a7fd2-38f9-4522-b994-73b7596e11c7" containerName="istio-proxy" Oct 30 06:19:14.173428 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.173147 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" Oct 30 06:19:14.173428 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.173155 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" Oct 30 06:19:14.176951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.176932 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.213873 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.213841 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d"] Oct 30 06:19:14.364837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.364805 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.364864 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365072 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365016 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365085 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365242 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365361 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnrl\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365497 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365482 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365602 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365588 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.365670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.365659 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.466976 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467063 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467159 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467235 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnrl\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467331 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467495 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467432 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467506 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467554 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467522 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467577 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467647 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.467809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.467784 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.468092 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.468072 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.468261 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.468084 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.468327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.468215 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.470611 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.470586 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.470728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.470615 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.492541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.492347 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnrl\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.493098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.493077 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:14.793301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:14.792453 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:15.127970 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:19:15.127943 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3ba20b_581d_4da1_8e15_ec98b13f08de.slice/crio-488e06aedbb121ef92942ea7f96a14b6f9cdbc44b1d3ae0916adc50ff0d304ce WatchSource:0}: Error finding container 488e06aedbb121ef92942ea7f96a14b6f9cdbc44b1d3ae0916adc50ff0d304ce: Status 404 returned error can't find the container with id 488e06aedbb121ef92942ea7f96a14b6f9cdbc44b1d3ae0916adc50ff0d304ce Oct 30 06:19:15.128521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:15.128501 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d"] Oct 30 06:19:15.131520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:15.131490 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:15.131616 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:15.131553 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:15.131616 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:15.131580 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:15.910338 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:15.910295 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" event={"ID":"9e3ba20b-581d-4da1-8e15-ec98b13f08de","Type":"ContainerStarted","Data":"c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c"} Oct 30 06:19:15.910879 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:15.910380 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" event={"ID":"9e3ba20b-581d-4da1-8e15-ec98b13f08de","Type":"ContainerStarted","Data":"488e06aedbb121ef92942ea7f96a14b6f9cdbc44b1d3ae0916adc50ff0d304ce"} Oct 30 06:19:16.000417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:16.000334 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podStartSLOduration=2.000312783 podStartE2EDuration="2.000312783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:15.974590973 +0000 UTC m=+2810.082630008" watchObservedRunningTime="2026-10-30 06:19:16.000312783 +0000 UTC m=+2810.108351817" Oct 30 06:19:16.795134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:16.794518 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:16.796130 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:16.796016 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:19:16.796130 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:16.796068 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:19:17.496136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.494085 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd"] Oct 30 06:19:17.511133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.509827 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb5598b-7e14-4635-be7a-8d80130a25ae" containerName="istio-proxy" Oct 30 06:19:17.511133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.509857 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc060397-36c4-4861-9c97-19eb24b3da3a" containerName="istio-proxy" Oct 30 06:19:17.515237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.515213 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd"] Oct 30 06:19:17.515397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.515352 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.535976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.535941 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.535994 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536126 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536211 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536350 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536334 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536464 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536515 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536648 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.536773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.536757 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t96c\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.638760 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.638725 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.638952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.638828 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t96c\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.638952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.638944 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.638975 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639005 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639075 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639046 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639142 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639223 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639261 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639560 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.639708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639659 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.640537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639733 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.640537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.639844 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.640537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.640522 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.641857 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.641831 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.642190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.642170 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.676127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.664627 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t96c\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.676127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.667535 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:17.812224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.795539 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:19:17.812224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.795614 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:19:17.836667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:17.836642 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:18.216400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:18.216369 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd"] Oct 30 06:19:18.221474 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:18.221438 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:18.221710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:18.221537 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:18.221710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:18.221583 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:18.799133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:18.797435 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:19:18.799133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:18.797490 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:19:19.021907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.021874 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" event={"ID":"268a9a7a-53a1-4d01-922f-4b444ba492b6","Type":"ContainerStarted","Data":"ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb"} Oct 30 06:19:19.022137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.021928 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" event={"ID":"268a9a7a-53a1-4d01-922f-4b444ba492b6","Type":"ContainerStarted","Data":"956da2c1bd10e286fe54ced9f606b75f936ae4ee02bbc8dcdb79083cf31923be"} Oct 30 06:19:19.798140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.795426 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:19:19.798140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.795498 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:19:19.837404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.837363 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:19.839057 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.839035 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 06:19:19.839182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:19.839081 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 06:19:20.659590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:20.656942 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:19:20.811133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:20.805696 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:19:20.811133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:20.805771 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:19:20.837176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:20.837146 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 06:19:20.837334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:20.837205 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 06:19:21.646229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:21.646193 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" podUID="ba6d5408-690c-4535-9740-64d45c107c05" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:19:21.799899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:21.799470 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:19:21.799899 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:21.799529 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:19:21.839208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:21.839170 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 06:19:21.839389 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:21.839234 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 06:19:22.823133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:22.822555 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:22.827128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:22.824288 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:22.851134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:22.831318 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:19:22.851134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:22.838927 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 06:19:22.851134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:22.838985 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 06:19:22.901763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:22.901697 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podStartSLOduration=5.901663588 podStartE2EDuration="5.901663588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:19.087446634 +0000 UTC m=+2813.195485668" watchObservedRunningTime="2026-10-30 06:19:22.901663588 +0000 UTC m=+2817.009702641" Oct 30 06:19:23.837845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:23.837806 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 06:19:23.838386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:23.837856 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 06:19:24.841557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:24.841521 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body= Oct 30 06:19:24.842298 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:24.842262 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" Oct 30 06:19:25.843240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:25.843211 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:25.844442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:25.844418 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:25.844982 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:25.844956 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:19:27.239773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.239367 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:19:27.290395 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.268726 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc\" (UniqueName: \"kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc\") pod \"5301a71a-b051-42fd-a14e-2ae51a411dd5\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " Oct 30 06:19:27.290395 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.269121 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume\") pod \"5301a71a-b051-42fd-a14e-2ae51a411dd5\" (UID: \"5301a71a-b051-42fd-a14e-2ae51a411dd5\") " Oct 30 06:19:27.290395 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.270243 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume" pod "5301a71a-b051-42fd-a14e-2ae51a411dd5" (UID: "5301a71a-b051-42fd-a14e-2ae51a411dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:27.352630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.352125 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc" pod "5301a71a-b051-42fd-a14e-2ae51a411dd5" (UID: "5301a71a-b051-42fd-a14e-2ae51a411dd5"). InnerVolumeSpecName "kube-api-access-6rfsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:27.354235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.354048 5980 generic.go:356] "Generic (PLEG): container finished" podID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerID="5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24" exitCode=0 Oct 30 06:19:27.354624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.354409 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" event={"ID":"5301a71a-b051-42fd-a14e-2ae51a411dd5","Type":"ContainerDied","Data":"5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24"} Oct 30 06:19:27.354624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.354463 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" event={"ID":"5301a71a-b051-42fd-a14e-2ae51a411dd5","Type":"ContainerDied","Data":"9e49f3e91cb4919680a625e4a56d85618083623a9d172250fcd5da3669c79195"} Oct 30 06:19:27.354624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.354496 5980 scope.go:122] "RemoveContainer" containerID="5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24" Oct 30 06:19:27.354796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.354626 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm" Oct 30 06:19:27.361334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.360985 5980 generic.go:356] "Generic (PLEG): container finished" podID="ba6d5408-690c-4535-9740-64d45c107c05" containerID="12e6a8948ffebe796189004c3f041f0b94f48f0dea34a53c6fdb4b001d8fc948" exitCode=0 Oct 30 06:19:27.361334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.361079 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" event={"ID":"ba6d5408-690c-4535-9740-64d45c107c05","Type":"ContainerDied","Data":"12e6a8948ffebe796189004c3f041f0b94f48f0dea34a53c6fdb4b001d8fc948"} Oct 30 06:19:27.372220 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.371914 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5301a71a-b051-42fd-a14e-2ae51a411dd5-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:27.372220 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.371944 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rfsc\" (UniqueName: \"kubernetes.io/projected/5301a71a-b051-42fd-a14e-2ae51a411dd5-kube-api-access-6rfsc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:27.438878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.438841 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:19:27.443405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.443382 5980 scope.go:122] "RemoveContainer" containerID="5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24" Oct 30 06:19:27.443736 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:19:27.443706 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24\": container with ID starting with 5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24 not found: ID does not exist" containerID="5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24" Oct 30 06:19:27.443822 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.443749 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24"} err="failed to get container status \"5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24\": rpc error: code = NotFound desc = could not find container \"5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24\": container with ID starting with 5fd8fb37a73395657387b1bf1418a3b96c0bef9ab8c3eb74c2c62957e5151a24 not found: ID does not exist" Oct 30 06:19:27.519949 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.519912 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw\" (UniqueName: \"kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw\") pod \"ba6d5408-690c-4535-9740-64d45c107c05\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " Oct 30 06:19:27.520255 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.520231 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume\") pod \"ba6d5408-690c-4535-9740-64d45c107c05\" (UID: \"ba6d5408-690c-4535-9740-64d45c107c05\") " Oct 30 06:19:27.526688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.526657 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume" pod "ba6d5408-690c-4535-9740-64d45c107c05" (UID: "ba6d5408-690c-4535-9740-64d45c107c05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:27.533316 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.532518 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm"] Oct 30 06:19:27.536377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.536355 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfe14f-544dc775f-sksfm"] Oct 30 06:19:27.544445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.544422 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw" pod "ba6d5408-690c-4535-9740-64d45c107c05" (UID: "ba6d5408-690c-4535-9740-64d45c107c05"). InnerVolumeSpecName "kube-api-access-btgjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:27.644756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.635012 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btgjw\" (UniqueName: \"kubernetes.io/projected/ba6d5408-690c-4535-9740-64d45c107c05-kube-api-access-btgjw\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:27.644756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:27.644547 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6d5408-690c-4535-9740-64d45c107c05-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:28.029843 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:28.029760 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" path="/var/lib/kubelet/pods/5301a71a-b051-42fd-a14e-2ae51a411dd5/volumes" Oct 30 06:19:28.127165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:28.126127 5980 scope.go:122] "RemoveContainer" containerID="12e6a8948ffebe796189004c3f041f0b94f48f0dea34a53c6fdb4b001d8fc948" Oct 30 06:19:28.384696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:28.384662 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" Oct 30 06:19:28.397366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:28.388179 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj" event={"ID":"ba6d5408-690c-4535-9740-64d45c107c05","Type":"ContainerDied","Data":"778c28ac04ccb28cbb4d3b06cfb483983dcacd60113ddda9d533a445fd536d08"} Oct 30 06:19:28.512138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:28.504207 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj"] Oct 30 06:19:28.528527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:28.528496 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-cfe14f-7cfd4494b7-wj6lj"] Oct 30 06:19:30.033503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:30.033468 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6d5408-690c-4535-9740-64d45c107c05" path="/var/lib/kubelet/pods/ba6d5408-690c-4535-9740-64d45c107c05/volumes" Oct 30 06:19:33.819564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:33.817232 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:19:33.819564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:33.817315 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:19:34.959724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:34.959684 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2"] Oct 30 06:19:34.994225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:34.994193 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4"] Oct 30 06:19:36.743524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:36.743482 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:19:36.744207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:36.743568 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:19:39.960503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:39.960207 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" podUID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" containerName="payload-processing" containerID="cri-o://04e09edd0a4336905d686ee003d5d41794e3999ec61106fb8d842779c07d21b0" gracePeriod=25 Oct 30 06:19:39.995036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:39.994990 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" podUID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" containerName="payload-pre-processing" containerID="cri-o://13321c9f757e5f5870eb67b1dc0a3150be4944b5bbbb3d4ce928f0a7d28108f5" gracePeriod=25 Oct 30 06:19:41.493138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.492333 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr"] Oct 30 06:19:41.493138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.492640 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" containerID="cri-o://646642e53ffc0108c477fc5b1d8e1dd0923d8f66f44b46e09d85b968e051bf29" gracePeriod=30 Oct 30 06:19:41.724170 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:19:41.723991 5980 logging.go:55] [core] [Channel #6773 SubChannel #6774]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.174:50052", ServerName: "10.134.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.174:50052: connect: connection refused" Oct 30 06:19:41.724170 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:19:41.724030 5980 logging.go:55] [core] [Channel #6775 SubChannel #6776]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.173:50052: connect: connection refused" Oct 30 06:19:41.877533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.877491 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-p64x4" podUID="b0f53d1a-6fce-4313-b6b9-3153e29f7f13" containerName="postgres" probeResult="failure" output="command timed out" Oct 30 06:19:41.984445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.984354 5980 generic.go:356] "Generic (PLEG): container finished" podID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" containerID="13321c9f757e5f5870eb67b1dc0a3150be4944b5bbbb3d4ce928f0a7d28108f5" exitCode=0 Oct 30 06:19:42.063307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.984428 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" event={"ID":"b05169d2-d0b4-4ee8-b746-ccc3f2df157f","Type":"ContainerDied","Data":"13321c9f757e5f5870eb67b1dc0a3150be4944b5bbbb3d4ce928f0a7d28108f5"} Oct 30 06:19:42.063307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.986296 5980 generic.go:356] "Generic (PLEG): container finished" podID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" containerID="04e09edd0a4336905d686ee003d5d41794e3999ec61106fb8d842779c07d21b0" exitCode=0 Oct 30 06:19:42.063307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:41.986372 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" event={"ID":"f08df514-f3c2-4c6e-9b36-b45e36d9440d","Type":"ContainerDied","Data":"04e09edd0a4336905d686ee003d5d41794e3999ec61106fb8d842779c07d21b0"} Oct 30 06:19:42.721183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:42.720735 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:42.721183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:42.720819 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:42.723961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:42.723774 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" podUID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded" Oct 30 06:19:42.723961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:42.723900 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" podUID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.174:50052\" within 1s: context deadline exceeded" Oct 30 06:19:43.294625 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:19:43.294563 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:19:43.548982 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.548924 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:19:43.617586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.617560 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:19:43.644612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.644580 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp\" (UniqueName: \"kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp\") pod \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " Oct 30 06:19:43.646630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.645547 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume\") pod \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\" (UID: \"f08df514-f3c2-4c6e-9b36-b45e36d9440d\") " Oct 30 06:19:43.647809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.646738 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume" pod "f08df514-f3c2-4c6e-9b36-b45e36d9440d" (UID: "f08df514-f3c2-4c6e-9b36-b45e36d9440d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:43.716390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.716317 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp" pod "f08df514-f3c2-4c6e-9b36-b45e36d9440d" (UID: "f08df514-f3c2-4c6e-9b36-b45e36d9440d"). InnerVolumeSpecName "kube-api-access-6bwxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:43.748069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.748041 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28\" (UniqueName: \"kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28\") pod \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " Oct 30 06:19:43.748587 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.748393 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume\") pod \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\" (UID: \"b05169d2-d0b4-4ee8-b746-ccc3f2df157f\") " Oct 30 06:19:43.748801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.748775 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume" pod "b05169d2-d0b4-4ee8-b746-ccc3f2df157f" (UID: "b05169d2-d0b4-4ee8-b746-ccc3f2df157f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:43.749213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.749191 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f08df514-f3c2-4c6e-9b36-b45e36d9440d-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:43.749311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.749218 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:43.749311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.749230 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bwxp\" (UniqueName: \"kubernetes.io/projected/f08df514-f3c2-4c6e-9b36-b45e36d9440d-kube-api-access-6bwxp\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:43.751489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.751462 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28" pod "b05169d2-d0b4-4ee8-b746-ccc3f2df157f" (UID: "b05169d2-d0b4-4ee8-b746-ccc3f2df157f"). InnerVolumeSpecName "kube-api-access-prv28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:43.850608 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:43.850579 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prv28\" (UniqueName: \"kubernetes.io/projected/b05169d2-d0b4-4ee8-b746-ccc3f2df157f-kube-api-access-prv28\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:44.006846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.006813 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" Oct 30 06:19:44.008456 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.008436 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" Oct 30 06:19:44.112563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.017261 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4" event={"ID":"b05169d2-d0b4-4ee8-b746-ccc3f2df157f","Type":"ContainerDied","Data":"11192538e524d1deaf8c208e82aab694c03c1b26809788c94fbfc009cda95341"} Oct 30 06:19:44.112563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.017289 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2" event={"ID":"f08df514-f3c2-4c6e-9b36-b45e36d9440d","Type":"ContainerDied","Data":"659929635b90fb66ad1bac1e00dd3478192a4daf0d0d9dfdce714715ead04239"} Oct 30 06:19:44.112563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.017322 5980 scope.go:122] "RemoveContainer" containerID="13321c9f757e5f5870eb67b1dc0a3150be4944b5bbbb3d4ce928f0a7d28108f5" Oct 30 06:19:44.112563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.070916 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4"] Oct 30 06:19:44.112563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.077431 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-79cd7bf999-nz6s4"] Oct 30 06:19:44.112563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.092431 5980 scope.go:122] "RemoveContainer" containerID="04e09edd0a4336905d686ee003d5d41794e3999ec61106fb8d842779c07d21b0" Oct 30 06:19:44.130037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.125176 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2"] Oct 30 06:19:44.130530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:44.130507 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d51cd8-76cbb8cf8f-nd7m2"] Oct 30 06:19:45.494551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.494509 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv"] Oct 30 06:19:45.510648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.510618 5980 cpu_manager.go:414] "containerMap: removing container" podUID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" containerName="payload-pre-processing" Oct 30 06:19:45.511414 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511390 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" containerName="payload-pre-processing" Oct 30 06:19:45.511634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511616 5980 cpu_manager.go:414] "containerMap: removing container" podUID="ba6d5408-690c-4535-9740-64d45c107c05" containerName="payload-pre-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511745 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6d5408-690c-4535-9740-64d45c107c05" containerName="payload-pre-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511840 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerName="payload-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511851 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerName="payload-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511873 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" containerName="payload-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.511880 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" containerName="payload-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.512089 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" containerName="payload-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.512123 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" containerName="payload-pre-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.512139 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6d5408-690c-4535-9740-64d45c107c05" containerName="payload-pre-processing" Oct 30 06:19:45.535878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.512156 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5301a71a-b051-42fd-a14e-2ae51a411dd5" containerName="payload-processing" Oct 30 06:19:45.572578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.572549 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv"] Oct 30 06:19:45.572812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.572790 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.683648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.683614 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.683814 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.683673 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2ld\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.683870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.683811 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.683914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.683887 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.684039 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.684021 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.684083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.684061 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.684155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.684100 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.684248 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.684217 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.684383 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.684305 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798329 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798409 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2ld\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798491 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798531 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798619 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798647 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798685 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798728 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.798807 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.805212 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.805588 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.806221 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.810175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.806552 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.833496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.811167 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.833496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.823543 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.833496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.823928 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.833496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.830776 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2ld\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.835375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.835352 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:45.948086 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:45.948054 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:46.019936 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:46.019905 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05169d2-d0b4-4ee8-b746-ccc3f2df157f" path="/var/lib/kubelet/pods/b05169d2-d0b4-4ee8-b746-ccc3f2df157f/volumes" Oct 30 06:19:46.030302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:46.020390 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08df514-f3c2-4c6e-9b36-b45e36d9440d" path="/var/lib/kubelet/pods/f08df514-f3c2-4c6e-9b36-b45e36d9440d/volumes" Oct 30 06:19:46.855236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:46.855205 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv"] Oct 30 06:19:47.042563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.042477 5980 generic.go:356] "Generic (PLEG): container finished" podID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerID="646642e53ffc0108c477fc5b1d8e1dd0923d8f66f44b46e09d85b968e051bf29" exitCode=0 Oct 30 06:19:47.042742 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.042562 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" event={"ID":"c53c62e9-60a6-44d4-94d1-9391b1b3a14a","Type":"ContainerDied","Data":"646642e53ffc0108c477fc5b1d8e1dd0923d8f66f44b46e09d85b968e051bf29"} Oct 30 06:19:47.042742 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.042621 5980 scope.go:122] "RemoveContainer" containerID="20460ffbf18a89958ceb88fb219f9982c443c5f492e794b51515649c1eebf3c4" Oct 30 06:19:47.162347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.162296 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:47.162537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.162404 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:47.162537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.162449 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:47.868243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:47.862885 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c"] Oct 30 06:19:48.067070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.066620 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:19:48.116959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.116922 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" event={"ID":"c53c62e9-60a6-44d4-94d1-9391b1b3a14a","Type":"ContainerDied","Data":"ac78945b5328240a21a76d8190067053392b6a21209cc1b84dc49581440dd996"} Oct 30 06:19:48.118097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.117323 5980 scope.go:122] "RemoveContainer" containerID="646642e53ffc0108c477fc5b1d8e1dd0923d8f66f44b46e09d85b968e051bf29" Oct 30 06:19:48.118097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.117501 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr" Oct 30 06:19:48.137616 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.136999 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" event={"ID":"950d117c-c5ac-4284-b761-270f957da6c1","Type":"ContainerStarted","Data":"1baaf49c15d5e947426c9df7d4f499c1e68c16a32676aac09eaf93f416c132c5"} Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.147702 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.147746 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.147826 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.147856 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.147915 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.147972 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.148193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.148048 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.148644 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.148694 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8\") pod \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\" (UID: \"c53c62e9-60a6-44d4-94d1-9391b1b3a14a\") " Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.150406 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.150618 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.150801 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.150927 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.151192 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.155255 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.156629 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "kube-api-access-cn6b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.161064 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:19:48.174439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.162645 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token" pod "c53c62e9-60a6-44d4-94d1-9391b1b3a14a" (UID: "c53c62e9-60a6-44d4-94d1-9391b1b3a14a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:48.251838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251805 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.251838 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251841 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251855 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251870 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251882 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251895 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251907 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251920 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.290488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.251934 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn6b8\" (UniqueName: \"kubernetes.io/projected/c53c62e9-60a6-44d4-94d1-9391b1b3a14a-kube-api-access-cn6b8\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:19:48.470148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.470079 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr"] Oct 30 06:19:48.477153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.477086 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-7jtxr"] Oct 30 06:19:48.537471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.537441 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm"] Oct 30 06:19:48.596299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.538508 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" Oct 30 06:19:48.596299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.538526 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" Oct 30 06:19:48.596299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.538580 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" Oct 30 06:19:48.596299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.538590 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" Oct 30 06:19:48.596299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.538783 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" Oct 30 06:19:48.596299 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.538807 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" containerName="istio-proxy" Oct 30 06:19:48.744331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.744260 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm"] Oct 30 06:19:48.744331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.744302 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr"] Oct 30 06:19:48.744555 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.744394 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.808957 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47rd\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809043 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809089 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809294 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809506 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809550 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809630 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809665 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.810858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.809706 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.833620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.833227 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:19:48.833620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.833286 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.920736 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.920808 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.920896 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.920930 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.920972 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.921083 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47rd\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.921170 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.921217 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.921316 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.921927 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.929893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.922592 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.932577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.930962 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.932577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.931483 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:48.932577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.931709 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.953057 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.955659 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.958225 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:48.958715 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47rd\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:49.120948 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:49.146051 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" event={"ID":"950d117c-c5ac-4284-b761-270f957da6c1","Type":"ContainerStarted","Data":"db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424"} Oct 30 06:19:49.356896 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:49.195397 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podStartSLOduration=4.19537734 podStartE2EDuration="4.19537734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:49.193419109 +0000 UTC m=+2843.301458144" watchObservedRunningTime="2026-10-30 06:19:49.19537734 +0000 UTC m=+2843.303416374" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:49.948971 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:49.951973 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:49.952012 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.019772 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53c62e9-60a6-44d4-94d1-9391b1b3a14a" path="/var/lib/kubelet/pods/c53c62e9-60a6-44d4-94d1-9391b1b3a14a/volumes" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.212295 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm"] Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.218801 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.218891 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.218949 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.949087 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:50.949169 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.199245 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" event={"ID":"fc7c8255-55fa-4331-b9cc-d66298842b37","Type":"ContainerStarted","Data":"1887a0cc290a062bd2077fabae32e7208bbdbd581c8d94eb042d794746a896fc"} Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.199349 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" event={"ID":"fc7c8255-55fa-4331-b9cc-d66298842b37","Type":"ContainerStarted","Data":"e668bcc7ac5069ce0388dbeddf70f877abc6f012f01100a9312b0f73e0969762"} Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.285609 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podStartSLOduration=3.285586854 podStartE2EDuration="3.285586854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:51.26196067 +0000 UTC m=+2845.369999702" watchObservedRunningTime="2026-10-30 06:19:51.285586854 +0000 UTC m=+2845.393625889" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.713245 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.713315 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.949147 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:51.949212 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:52.126035 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:52.133327 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:52.133387 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:53.023294 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:53.023366 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:53.131715 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:53.131780 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:53.949145 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.073256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:53.949187 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:19:54.151409 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.144051 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.151409 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.144122 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:19:54.329561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.291062 5980 generic.go:356] "Generic (PLEG): container finished" podID="377612ce-6b65-4347-949f-d03e31a4c596" containerID="30107178ffb80b3edd9b8257918d65a04c5835ca07e68ca109bcd14ed6bed061" exitCode=0 Oct 30 06:19:54.329561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.291238 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" event={"ID":"377612ce-6b65-4347-949f-d03e31a4c596","Type":"ContainerDied","Data":"30107178ffb80b3edd9b8257918d65a04c5835ca07e68ca109bcd14ed6bed061"} Oct 30 06:19:54.329561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.300747 5980 scope.go:122] "RemoveContainer" containerID="30107178ffb80b3edd9b8257918d65a04c5835ca07e68ca109bcd14ed6bed061" Oct 30 06:19:54.329561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.315255 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:54.329561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.315359 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:54.329561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.315435 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:54.949549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.949467 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.949549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:54.949528 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:19:55.130632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.130177 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:19:55.130632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.130237 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:19:55.336190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.333321 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" event={"ID":"377612ce-6b65-4347-949f-d03e31a4c596","Type":"ContainerStarted","Data":"c62c9fd49d351281d0c9605d104dd2fcb35967bc043f4fad89fd60eacaccc81d"} Oct 30 06:19:55.440921 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.440892 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:19:55.441219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.441086 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:19:55.441402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.441377 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:19:55.954712 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.954676 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:55.955489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.955468 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:55.957081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:55.956072 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:19:56.142262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:56.142226 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:19:56.142753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:56.142281 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:19:56.441044 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:56.441009 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:19:56.441342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:56.441315 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:19:57.155046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:57.152239 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:19:57.155046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:57.152322 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:19:57.441963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:57.441884 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:19:57.441963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:57.441943 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:19:58.182687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:58.178816 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:58.182687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:58.180214 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:58.199069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:58.198299 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:19:58.441416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:58.441344 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:19:58.441416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:58.441397 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:19:59.475277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:59.475245 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:19:59.548348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:19:59.480178 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:20:00.444910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:00.444467 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:20:00.444910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:00.444533 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:20:01.441939 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.441904 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:20:01.442472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.441966 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:20:01.519856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.519826 5980 generic.go:356] "Generic (PLEG): container finished" podID="47573111-9bc0-43e8-b490-f6d908e928fe" containerID="ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0" exitCode=0 Oct 30 06:20:01.520035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.519892 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" event={"ID":"47573111-9bc0-43e8-b490-f6d908e928fe","Type":"ContainerDied","Data":"ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0"} Oct 30 06:20:01.520376 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.520359 5980 scope.go:122] "RemoveContainer" containerID="ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0" Oct 30 06:20:01.522265 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.522235 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:01.522371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.522311 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:01.522371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:01.522344 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:02.542314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:02.542281 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:20:02.543862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:02.543830 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:20:02.546334 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:02.546302 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:20:02.552410 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:02.552348 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" event={"ID":"47573111-9bc0-43e8-b490-f6d908e928fe","Type":"ContainerStarted","Data":"f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e"} Oct 30 06:20:03.483133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:03.482304 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" podUID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" containerName="main" containerID="cri-o://28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472" gracePeriod=45 Oct 30 06:20:03.497651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:03.488426 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:20:03.497973 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:03.497947 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 06:20:03.498189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:03.498156 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 06:20:03.806842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:03.806803 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" podUID="2a2a0442-d5e7-4823-9ad3-11c53b921113" containerName="main" containerID="cri-o://f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df" gracePeriod=45 Oct 30 06:20:03.925444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:03.925411 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" podUID="2a2a0442-d5e7-4823-9ad3-11c53b921113" containerName="main" probeResult="failure" output="Get \"https://10.134.0.177:8000/ready\": dial tcp 10.134.0.177:8000: connect: connection refused" Oct 30 06:20:04.086185 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.060531 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" podUID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": dial tcp 10.134.0.170:8000: connect: connection refused" Oct 30 06:20:04.226993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.226971 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:20:04.310475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.310436 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home\") pod \"2a2a0442-d5e7-4823-9ad3-11c53b921113\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " Oct 30 06:20:04.310638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.310546 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs\") pod \"2a2a0442-d5e7-4823-9ad3-11c53b921113\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " Oct 30 06:20:04.310683 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.310632 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2\" (UniqueName: \"kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2\") pod \"2a2a0442-d5e7-4823-9ad3-11c53b921113\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " Oct 30 06:20:04.310719 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.310682 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm\") pod \"2a2a0442-d5e7-4823-9ad3-11c53b921113\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " Oct 30 06:20:04.310884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.310866 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir\") pod \"2a2a0442-d5e7-4823-9ad3-11c53b921113\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " Oct 30 06:20:04.311438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.310918 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache\") pod \"2a2a0442-d5e7-4823-9ad3-11c53b921113\" (UID: \"2a2a0442-d5e7-4823-9ad3-11c53b921113\") " Oct 30 06:20:04.312186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.312157 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache" pod "2a2a0442-d5e7-4823-9ad3-11c53b921113" (UID: "2a2a0442-d5e7-4823-9ad3-11c53b921113"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.312475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.312452 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home" pod "2a2a0442-d5e7-4823-9ad3-11c53b921113" (UID: "2a2a0442-d5e7-4823-9ad3-11c53b921113"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.313885 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.313805 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir" pod "2a2a0442-d5e7-4823-9ad3-11c53b921113" (UID: "2a2a0442-d5e7-4823-9ad3-11c53b921113"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.321204 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.321168 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs" pod "2a2a0442-d5e7-4823-9ad3-11c53b921113" (UID: "2a2a0442-d5e7-4823-9ad3-11c53b921113"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:04.322990 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.321723 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2" pod "2a2a0442-d5e7-4823-9ad3-11c53b921113" (UID: "2a2a0442-d5e7-4823-9ad3-11c53b921113"). InnerVolumeSpecName "kube-api-access-jv4v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:04.323269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.323243 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm" pod "2a2a0442-d5e7-4823-9ad3-11c53b921113" (UID: "2a2a0442-d5e7-4823-9ad3-11c53b921113"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.423191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.420956 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.423191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.420998 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.423191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.421013 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.423191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.421026 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a2a0442-d5e7-4823-9ad3-11c53b921113-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.423191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.421039 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv4v2\" (UniqueName: \"kubernetes.io/projected/2a2a0442-d5e7-4823-9ad3-11c53b921113-kube-api-access-jv4v2\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.423191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.421052 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a2a0442-d5e7-4823-9ad3-11c53b921113-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.488501 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.488461 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 06:20:04.488709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.488526 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 06:20:04.521941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.521912 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:20:04.596499 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.596359 5980 generic.go:356] "Generic (PLEG): container finished" podID="2a2a0442-d5e7-4823-9ad3-11c53b921113" containerID="f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df" exitCode=0 Oct 30 06:20:04.598016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.597984 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" event={"ID":"2a2a0442-d5e7-4823-9ad3-11c53b921113","Type":"ContainerDied","Data":"f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df"} Oct 30 06:20:04.598174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.598042 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" event={"ID":"2a2a0442-d5e7-4823-9ad3-11c53b921113","Type":"ContainerDied","Data":"890a3d3411dc764d3812e76ac894be26c15e8c3ef5eac1a81ee034352f272c0a"} Oct 30 06:20:04.598174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.598166 5980 scope.go:122] "RemoveContainer" containerID="f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df" Oct 30 06:20:04.598980 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.598958 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr" Oct 30 06:20:04.606139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.606080 5980 generic.go:356] "Generic (PLEG): container finished" podID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" containerID="28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472" exitCode=0 Oct 30 06:20:04.607067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.606296 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" Oct 30 06:20:04.607067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.606398 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" event={"ID":"8fc9fc10-b4da-4639-90da-9c6661ea2dbc","Type":"ContainerDied","Data":"28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472"} Oct 30 06:20:04.607067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.606452 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c" event={"ID":"8fc9fc10-b4da-4639-90da-9c6661ea2dbc","Type":"ContainerDied","Data":"5b5524e95bea722b38a059068d038dc94b1651f1a91253a2e306bb3fde5c7f6a"} Oct 30 06:20:04.628284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.628256 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home\") pod \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " Oct 30 06:20:04.628405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.628334 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm\") pod \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " Oct 30 06:20:04.628468 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.628449 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc\" (UniqueName: \"kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc\") pod \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " Oct 30 06:20:04.628524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.628486 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs\") pod \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " Oct 30 06:20:04.628579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.628559 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache\") pod \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " Oct 30 06:20:04.628695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.628678 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir\") pod \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\" (UID: \"8fc9fc10-b4da-4639-90da-9c6661ea2dbc\") " Oct 30 06:20:04.631256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.631232 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home" pod "8fc9fc10-b4da-4639-90da-9c6661ea2dbc" (UID: "8fc9fc10-b4da-4639-90da-9c6661ea2dbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.632015 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.631924 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache" pod "8fc9fc10-b4da-4639-90da-9c6661ea2dbc" (UID: "8fc9fc10-b4da-4639-90da-9c6661ea2dbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.635846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.635825 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm" pod "8fc9fc10-b4da-4639-90da-9c6661ea2dbc" (UID: "8fc9fc10-b4da-4639-90da-9c6661ea2dbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.639096 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.638337 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir" pod "8fc9fc10-b4da-4639-90da-9c6661ea2dbc" (UID: "8fc9fc10-b4da-4639-90da-9c6661ea2dbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:04.640282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.640259 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc" pod "8fc9fc10-b4da-4639-90da-9c6661ea2dbc" (UID: "8fc9fc10-b4da-4639-90da-9c6661ea2dbc"). InnerVolumeSpecName "kube-api-access-v6hlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:04.641965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.641941 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs" pod "8fc9fc10-b4da-4639-90da-9c6661ea2dbc" (UID: "8fc9fc10-b4da-4639-90da-9c6661ea2dbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:04.643675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.643654 5980 scope.go:122] "RemoveContainer" containerID="f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df" Oct 30 06:20:04.645223 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:04.644334 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df\": container with ID starting with f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df not found: ID does not exist" containerID="f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df" Oct 30 06:20:04.645223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.644390 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df"} err="failed to get container status \"f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df\": rpc error: code = NotFound desc = could not find container \"f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df\": container with ID starting with f3864692c09ad3713af94cfa3948aa120b3c0aa57a1c27ec7a457feaed2a50df not found: ID does not exist" Oct 30 06:20:04.645223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.644415 5980 scope.go:122] "RemoveContainer" containerID="28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472" Oct 30 06:20:04.660459 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.660438 5980 scope.go:122] "RemoveContainer" containerID="28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472" Oct 30 06:20:04.660792 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:04.660768 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472\": container with ID starting with 28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472 not found: ID does not exist" containerID="28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472" Oct 30 06:20:04.660877 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.660803 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472"} err="failed to get container status \"28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472\": rpc error: code = NotFound desc = could not find container \"28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472\": container with ID starting with 28bbe7a68ba50ea8f3077b0d5fd6bf8cac2a57897bbd6b6c0a2f9b61858ec472 not found: ID does not exist" Oct 30 06:20:04.707318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.707195 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr"] Oct 30 06:20:04.709567 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.709545 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/rate-test-model-91f42c-kserve-6db4f58f69-pvgmr"] Oct 30 06:20:04.732866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.732844 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6hlc\" (UniqueName: \"kubernetes.io/projected/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-kube-api-access-v6hlc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.732866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.732867 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.733511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.732878 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.733511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.732886 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.733511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.732893 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.733511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.732900 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc9fc10-b4da-4639-90da-9c6661ea2dbc-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.959660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.959590 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c"] Oct 30 06:20:04.992135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:04.991402 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/rate-test-model-d1bfce-kserve-574d7c78cc-v2x9c"] Oct 30 06:20:05.488131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:05.487443 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 06:20:05.488131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:05.487516 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 06:20:05.684140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:05.683214 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:20:06.018813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:06.018779 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2a0442-d5e7-4823-9ad3-11c53b921113" path="/var/lib/kubelet/pods/2a2a0442-d5e7-4823-9ad3-11c53b921113/volumes" Oct 30 06:20:06.019335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:06.019318 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" path="/var/lib/kubelet/pods/8fc9fc10-b4da-4639-90da-9c6661ea2dbc/volumes" Oct 30 06:20:06.494141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:06.493639 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 06:20:06.494141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:06.493715 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 06:20:07.493139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:07.488732 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 06:20:07.493139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:07.488813 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 06:20:08.521234 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:08.519916 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:20:08.524566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:08.522199 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:20:13.110925 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:13.110886 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:20:14.309312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:14.309272 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29"] Oct 30 06:20:14.309812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:14.309484 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" containerID="cri-o://3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12" gracePeriod=30 Oct 30 06:20:15.484657 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:15.484618 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb"] Oct 30 06:20:15.485203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:15.484880 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" containerID="cri-o://a44e9bf9a8be95eb1a8f759d0a12696431cfaa6c13ec701257898fd7cb312723" gracePeriod=30 Oct 30 06:20:19.697170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.697140 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:20:19.729286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.729237 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.729451 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.729373 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.729560 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.729504 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:19.729691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.729669 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:19.730269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.729811 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.730269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730031 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:19.730269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730088 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730487 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730564 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730628 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730751 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730861 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data\") pod \"478a125a-bb6b-4338-9d02-5904cb583511\" (UID: \"478a125a-bb6b-4338-9d02-5904cb583511\") " Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.730950 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.731336 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.731857 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/478a125a-bb6b-4338-9d02-5904cb583511-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.731878 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.731891 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.731905 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.732251 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.731917 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.734082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.732896 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:19.734082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.733164 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:19.734082 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.733747 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:19.734550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.734526 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v" pod "478a125a-bb6b-4338-9d02-5904cb583511" (UID: "478a125a-bb6b-4338-9d02-5904cb583511"). InnerVolumeSpecName "kube-api-access-d6r8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:19.741168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.741139 5980 generic.go:356] "Generic (PLEG): container finished" podID="478a125a-bb6b-4338-9d02-5904cb583511" containerID="3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12" exitCode=0 Oct 30 06:20:19.741287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.741200 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" Oct 30 06:20:19.741287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.741222 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" event={"ID":"478a125a-bb6b-4338-9d02-5904cb583511","Type":"ContainerDied","Data":"3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12"} Oct 30 06:20:19.741427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.741283 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29" event={"ID":"478a125a-bb6b-4338-9d02-5904cb583511","Type":"ContainerDied","Data":"bc7d20b82656dcb4e01373f0e45f691279ed703d75ca8991316c6e646acb101e"} Oct 30 06:20:19.741427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.741317 5980 scope.go:122] "RemoveContainer" containerID="3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12" Oct 30 06:20:19.776707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.776681 5980 scope.go:122] "RemoveContainer" containerID="3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12" Oct 30 06:20:19.777754 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:19.777714 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12\": container with ID starting with 3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12 not found: ID does not exist" containerID="3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12" Oct 30 06:20:19.777964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.777758 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12"} err="failed to get container status \"3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12\": rpc error: code = NotFound desc = could not find container \"3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12\": container with ID starting with 3e57c589c164797ace20ef8fb37b98680329e91385e0f3689ebc808cce2daf12 not found: ID does not exist" Oct 30 06:20:19.824051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.824017 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29"] Oct 30 06:20:19.829959 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.829933 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-q5p29"] Oct 30 06:20:19.835842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.835809 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6r8v\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-kube-api-access-d6r8v\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.835969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.835847 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/478a125a-bb6b-4338-9d02-5904cb583511-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.835969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.835864 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/478a125a-bb6b-4338-9d02-5904cb583511-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:19.835969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:19.835908 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/478a125a-bb6b-4338-9d02-5904cb583511-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.020980 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.020898 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478a125a-bb6b-4338-9d02-5904cb583511" path="/var/lib/kubelet/pods/478a125a-bb6b-4338-9d02-5904cb583511/volumes" Oct 30 06:20:20.764238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.764157 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv"] Oct 30 06:20:20.765040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.765008 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" containerID="cri-o://7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662" gracePeriod=30 Oct 30 06:20:20.770663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.768941 5980 generic.go:356] "Generic (PLEG): container finished" podID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerID="a44e9bf9a8be95eb1a8f759d0a12696431cfaa6c13ec701257898fd7cb312723" exitCode=0 Oct 30 06:20:20.770663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.769906 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" event={"ID":"d5ba055a-3427-42e5-b30a-9c1d89dcf790","Type":"ContainerDied","Data":"a44e9bf9a8be95eb1a8f759d0a12696431cfaa6c13ec701257898fd7cb312723"} Oct 30 06:20:20.886616 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.886589 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979443 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979557 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979638 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979679 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979749 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979789 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979824 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.979875 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.981271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.980006 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket\") pod \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\" (UID: \"d5ba055a-3427-42e5-b30a-9c1d89dcf790\") " Oct 30 06:20:20.982654 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.982347 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.983741 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.983719 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.983883 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.983781 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.984027 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.983996 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.984390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.984358 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:21.000323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:20.998305 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:21.003812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.003741 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "kube-api-access-4vz67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:21.006135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.005875 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:21.013734 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.013694 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy" pod "d5ba055a-3427-42e5-b30a-9c1d89dcf790" (UID: "d5ba055a-3427-42e5-b30a-9c1d89dcf790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:21.083739 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.083703 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.083974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.083955 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.083979 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.083993 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.084006 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.084020 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.084032 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ba055a-3427-42e5-b30a-9c1d89dcf790-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.084044 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vz67\" (UniqueName: \"kubernetes.io/projected/d5ba055a-3427-42e5-b30a-9c1d89dcf790-kube-api-access-4vz67\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.084549 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.084054 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ba055a-3427-42e5-b30a-9c1d89dcf790-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:21.782787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.782702 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" Oct 30 06:20:21.782787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.782714 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb" event={"ID":"d5ba055a-3427-42e5-b30a-9c1d89dcf790","Type":"ContainerDied","Data":"29f45ffe79dbd71134dae0471ff33dcb005d9dbccb06afb380a6e86e0483b564"} Oct 30 06:20:21.783363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.782787 5980 scope.go:122] "RemoveContainer" containerID="a44e9bf9a8be95eb1a8f759d0a12696431cfaa6c13ec701257898fd7cb312723" Oct 30 06:20:21.913136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.912582 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb"] Oct 30 06:20:21.946128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:21.940021 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-65fzb"] Oct 30 06:20:22.043180 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:22.042297 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" path="/var/lib/kubelet/pods/d5ba055a-3427-42e5-b30a-9c1d89dcf790/volumes" Oct 30 06:20:23.132929 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:23.132901 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z"] Oct 30 06:20:23.133377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:23.133099 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" containerID="cri-o://7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f" gracePeriod=30 Oct 30 06:20:25.296711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:25.296679 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-7dz5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:20:25.297152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:25.296768 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:26.143246 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.143216 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:20:26.256569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.256532 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.257159 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.257125 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:26.257901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.257872 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.258275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258229 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258297 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258334 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258366 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258728 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258796 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.258916 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.259048 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259513 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.259093 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx\") pod \"403e3648-157f-404c-b8d2-42204133c0a7\" (UID: \"403e3648-157f-404c-b8d2-42204133c0a7\") " Oct 30 06:20:26.259649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.259625 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:26.260052 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.260023 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/403e3648-157f-404c-b8d2-42204133c0a7-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.260224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.260055 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.260224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.260070 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.260224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.260083 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.260657 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.260634 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:26.265738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.265707 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:26.266571 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.266546 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:26.267598 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.267538 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:26.268311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.268288 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx" pod "403e3648-157f-404c-b8d2-42204133c0a7" (UID: "403e3648-157f-404c-b8d2-42204133c0a7"). InnerVolumeSpecName "kube-api-access-qh4bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:26.362051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.361969 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/403e3648-157f-404c-b8d2-42204133c0a7-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.362051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.361999 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.362051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.362009 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/403e3648-157f-404c-b8d2-42204133c0a7-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.362051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.362019 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh4bx\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-kube-api-access-qh4bx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.362051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.362032 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/403e3648-157f-404c-b8d2-42204133c0a7-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:26.837864 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.837833 5980 generic.go:356] "Generic (PLEG): container finished" podID="403e3648-157f-404c-b8d2-42204133c0a7" containerID="7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662" exitCode=0 Oct 30 06:20:26.838077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.837935 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" event={"ID":"403e3648-157f-404c-b8d2-42204133c0a7","Type":"ContainerDied","Data":"7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662"} Oct 30 06:20:26.838077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.837951 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" Oct 30 06:20:26.838077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.837982 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv" event={"ID":"403e3648-157f-404c-b8d2-42204133c0a7","Type":"ContainerDied","Data":"b47ddd405167e9306486ef4a4008735caebf49924fcceb8f41d14e6f78e105c4"} Oct 30 06:20:26.838077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.838005 5980 scope.go:122] "RemoveContainer" containerID="7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662" Oct 30 06:20:26.866101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.866078 5980 scope.go:122] "RemoveContainer" containerID="7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662" Oct 30 06:20:26.866530 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:26.866501 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662\": container with ID starting with 7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662 not found: ID does not exist" containerID="7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662" Oct 30 06:20:26.866630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.866539 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662"} err="failed to get container status \"7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662\": rpc error: code = NotFound desc = could not find container \"7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662\": container with ID starting with 7b518cce999b61eb5dbea03c7eef90a5c9a0277d1dc6eebeeaeb22751ea46662 not found: ID does not exist" Oct 30 06:20:26.914491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.914456 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv"] Oct 30 06:20:26.926975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:26.926949 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-c8mkv"] Oct 30 06:20:28.014787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.014754 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403e3648-157f-404c-b8d2-42204133c0a7" path="/var/lib/kubelet/pods/403e3648-157f-404c-b8d2-42204133c0a7/volumes" Oct 30 06:20:28.482967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.482930 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 06:20:28.600034 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600007 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600089 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600211 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600293 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600441 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600410 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600450 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600495 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600517 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.600630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.600564 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket\") pod \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\" (UID: \"74b5d1f1-2dfd-42c8-ab76-e5305ae134df\") " Oct 30 06:20:28.601336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.601305 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.601483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.601321 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:28.602088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.602058 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.602392 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.602139 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.602392 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.602188 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.604530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.604487 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "kube-api-access-4x7qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:28.605151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.605076 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:28.605151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.605132 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:28.605954 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.605471 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo" pod "74b5d1f1-2dfd-42c8-ab76-e5305ae134df" (UID: "74b5d1f1-2dfd-42c8-ab76-e5305ae134df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:28.702765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702730 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.702765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702761 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7qx\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-kube-api-access-4x7qx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.702765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702771 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.703081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702780 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.703081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702789 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.703081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702797 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.703081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702804 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.703081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702811 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.703081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.702818 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74b5d1f1-2dfd-42c8-ab76-e5305ae134df-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:28.858297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.858202 5980 generic.go:356] "Generic (PLEG): container finished" podID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerID="7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f" exitCode=0 Oct 30 06:20:28.858297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.858236 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" event={"ID":"74b5d1f1-2dfd-42c8-ab76-e5305ae134df","Type":"ContainerDied","Data":"7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f"} Oct 30 06:20:28.858297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.858285 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" Oct 30 06:20:28.858523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.858360 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z" event={"ID":"74b5d1f1-2dfd-42c8-ab76-e5305ae134df","Type":"ContainerDied","Data":"6911e41a5e7e7549f9c55ddd7afbc09080dc41fbfe724ee87d5fc16a6fdb7ae4"} Oct 30 06:20:28.858523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.858391 5980 scope.go:122] "RemoveContainer" containerID="7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f" Oct 30 06:20:28.870856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.870840 5980 scope.go:122] "RemoveContainer" containerID="7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f" Oct 30 06:20:28.871204 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:28.871173 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f\": container with ID starting with 7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f not found: ID does not exist" containerID="7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f" Oct 30 06:20:28.871358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.871214 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f"} err="failed to get container status \"7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f\": rpc error: code = NotFound desc = could not find container \"7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f\": container with ID starting with 7c3a4d7753bb430343448d1ad35dd6106b49b174138323cb0d0938904de8d22f not found: ID does not exist" Oct 30 06:20:28.915757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.915727 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z"] Oct 30 06:20:28.928279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:28.928258 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-7dz5z"] Oct 30 06:20:30.016673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.016640 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" path="/var/lib/kubelet/pods/74b5d1f1-2dfd-42c8-ab76-e5305ae134df/volumes" Oct 30 06:20:30.625325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.625291 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6"] Oct 30 06:20:30.626306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626290 5980 cpu_manager.go:414] "containerMap: removing container" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" Oct 30 06:20:30.626353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626310 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" Oct 30 06:20:30.626385 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626359 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" containerName="main" Oct 30 06:20:30.626385 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626367 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" containerName="main" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626386 5980 cpu_manager.go:414] "containerMap: removing container" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626393 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626406 5980 cpu_manager.go:414] "containerMap: removing container" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626413 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626426 5980 cpu_manager.go:414] "containerMap: removing container" podUID="2a2a0442-d5e7-4823-9ad3-11c53b921113" containerName="main" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626432 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2a0442-d5e7-4823-9ad3-11c53b921113" containerName="main" Oct 30 06:20:30.626452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626447 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626454 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626610 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b5d1f1-2dfd-42c8-ab76-e5305ae134df" containerName="istio-proxy" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626624 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc9fc10-b4da-4639-90da-9c6661ea2dbc" containerName="main" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626637 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="403e3648-157f-404c-b8d2-42204133c0a7" containerName="istio-proxy" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626648 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="478a125a-bb6b-4338-9d02-5904cb583511" containerName="istio-proxy" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626656 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2a0442-d5e7-4823-9ad3-11c53b921113" containerName="main" Oct 30 06:20:30.627783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.626666 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ba055a-3427-42e5-b30a-9c1d89dcf790" containerName="istio-proxy" Oct 30 06:20:30.632421 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.632398 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.655717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.653314 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6"] Oct 30 06:20:30.737019 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.736981 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9xc\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737042 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737164 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737256 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737313 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737461 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737516 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737588 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737574 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.737644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.737631 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.839941 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840042 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840177 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840228 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840266 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840312 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840475 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9xc\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840518 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.840583 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.841304 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.841633 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.841935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.841923 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.842779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.841946 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.842779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.842165 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.844830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.844808 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.845415 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.845382 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.879280 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.879220 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.879388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.879352 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9xc\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:30.960283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:30.960252 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:31.289447 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:20:31.289419 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921ed1c2_f38e_4842_8fa7_104af4a9330a.slice/crio-f22ea79638905297df157ff1260769c48d678cc14339bb7bc6669bb2b59863c2 WatchSource:0}: Error finding container f22ea79638905297df157ff1260769c48d678cc14339bb7bc6669bb2b59863c2: Status 404 returned error can't find the container with id f22ea79638905297df157ff1260769c48d678cc14339bb7bc6669bb2b59863c2 Oct 30 06:20:31.290783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.290754 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6"] Oct 30 06:20:31.292483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.292438 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.292572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.292514 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.292572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.292546 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.893849 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.892340 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" event={"ID":"921ed1c2-f38e-4842-8fa7-104af4a9330a","Type":"ContainerStarted","Data":"f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5"} Oct 30 06:20:31.893849 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.892411 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" event={"ID":"921ed1c2-f38e-4842-8fa7-104af4a9330a","Type":"ContainerStarted","Data":"f22ea79638905297df157ff1260769c48d678cc14339bb7bc6669bb2b59863c2"} Oct 30 06:20:31.959152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.958958 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podStartSLOduration=1.958939531 podStartE2EDuration="1.958939531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:31.955585914 +0000 UTC m=+2886.063624959" watchObservedRunningTime="2026-10-30 06:20:31.958939531 +0000 UTC m=+2886.066978561" Oct 30 06:20:31.960385 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:31.960361 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:33.031556 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.029951 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:20:33.031556 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.030068 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 06:20:33.704529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.704489 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9"] Oct 30 06:20:33.710041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.710013 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.724608 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.724651 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.724768 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvnn\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.724866 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.724892 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.725006 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.725084 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.725143 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.726749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.725195 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.735021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.734993 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9"] Oct 30 06:20:33.826237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826205 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826241 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826294 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvnn\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826385 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826412 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826525 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826605 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826641 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826689 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826894 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826868 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.826962 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.826904 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.827039 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.827016 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.827138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.827062 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.827477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.827448 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.828983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.828953 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.829275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.829256 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.852727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.852696 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvnn\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.853183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.853160 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:33.961089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.961005 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:20:33.961089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:33.961062 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 06:20:34.033774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.033742 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:34.401294 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:20:34.401263 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f52cb4_299f_465f_a8aa_c980be86bbfc.slice/crio-e46792f48091950f97c5c6039009da1eac6188209e9b7cb02598448d20059428 WatchSource:0}: Error finding container e46792f48091950f97c5c6039009da1eac6188209e9b7cb02598448d20059428: Status 404 returned error can't find the container with id e46792f48091950f97c5c6039009da1eac6188209e9b7cb02598448d20059428 Oct 30 06:20:34.402062 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.402016 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9"] Oct 30 06:20:34.404285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.404250 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:34.404364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.404320 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:34.404364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.404349 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:34.961093 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.961059 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:20:34.961298 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:34.961129 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 06:20:35.048226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:35.048186 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" event={"ID":"26f52cb4-299f-465f-a8aa-c980be86bbfc","Type":"ContainerStarted","Data":"4b37efee3062c2cc3d9d45f873ca29545df4e001fbb681975ca67a462e4d43aa"} Oct 30 06:20:35.048695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:35.048249 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" event={"ID":"26f52cb4-299f-465f-a8aa-c980be86bbfc","Type":"ContainerStarted","Data":"e46792f48091950f97c5c6039009da1eac6188209e9b7cb02598448d20059428"} Oct 30 06:20:35.970189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:35.963297 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:20:35.970189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:35.963366 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 06:20:36.115195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:36.104548 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:20:36.115195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:36.104610 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:20:36.215143 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:36.211595 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:36.961038 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:36.960995 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:20:36.961341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:36.961055 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 06:20:37.034777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:37.034744 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.035002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:37.034813 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:20:37.960621 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:37.960579 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.961148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:37.960646 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 06:20:38.035226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:38.035188 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:20:38.035391 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:38.035255 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:20:38.975438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:38.975408 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:38.977570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:38.977547 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:38.984330 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:38.984310 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:20:39.034609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:39.034566 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:20:39.034791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:39.034635 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:20:39.044884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:39.044839 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podStartSLOduration=6.044826471 podStartE2EDuration="6.044826471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:35.108579156 +0000 UTC m=+2889.216618238" watchObservedRunningTime="2026-10-30 06:20:39.044826471 +0000 UTC m=+2893.152865559" Oct 30 06:20:40.034614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:40.034583 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:20:40.035089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:40.034647 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:20:41.034310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:41.034277 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:20:41.034514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:41.034335 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:20:42.039193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:42.039165 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:42.039650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:42.039630 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:42.040456 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:42.040438 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:20:43.092368 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:43.092330 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:20:44.169068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.169036 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks"] Oct 30 06:20:44.169850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.169817 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" podUID="76c9f39c-2e68-41a5-a7ab-95d3bb40e653" containerName="maas-api" containerID="cri-o://7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd" gracePeriod=30 Oct 30 06:20:44.201894 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.201864 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw"] Oct 30 06:20:44.202347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.202316 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" containerID="cri-o://3c782c1ce45a2bd66c0c92cd7fe65ea805f1c59448895a6f5ffe5d41ed15bfff" gracePeriod=30 Oct 30 06:20:44.760702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.760678 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:20:44.772590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.772565 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\") pod \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " Oct 30 06:20:44.772775 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.772698 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca\") pod \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " Oct 30 06:20:44.772844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.772795 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs\") pod \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " Oct 30 06:20:44.772844 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.772824 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx\" (UniqueName: \"kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx\") pod \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\" (UID: \"76c9f39c-2e68-41a5-a7ab-95d3bb40e653\") " Oct 30 06:20:44.773050 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.773023 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca" pod "76c9f39c-2e68-41a5-a7ab-95d3bb40e653" (UID: "76c9f39c-2e68-41a5-a7ab-95d3bb40e653"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:44.773843 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.773819 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:44.775264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.775243 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls" pod "76c9f39c-2e68-41a5-a7ab-95d3bb40e653" (UID: "76c9f39c-2e68-41a5-a7ab-95d3bb40e653"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:44.775524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.775498 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx" pod "76c9f39c-2e68-41a5-a7ab-95d3bb40e653" (UID: "76c9f39c-2e68-41a5-a7ab-95d3bb40e653"). InnerVolumeSpecName "kube-api-access-c5tmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:44.775738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.775719 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs" pod "76c9f39c-2e68-41a5-a7ab-95d3bb40e653" (UID: "76c9f39c-2e68-41a5-a7ab-95d3bb40e653"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:44.875700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.875669 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:44.875700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.875700 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:44.875893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:44.875713 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5tmx\" (UniqueName: \"kubernetes.io/projected/76c9f39c-2e68-41a5-a7ab-95d3bb40e653-kube-api-access-c5tmx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.329285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.328581 5980 generic.go:356] "Generic (PLEG): container finished" podID="76c9f39c-2e68-41a5-a7ab-95d3bb40e653" containerID="7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd" exitCode=0 Oct 30 06:20:45.329285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.328736 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" event={"ID":"76c9f39c-2e68-41a5-a7ab-95d3bb40e653","Type":"ContainerDied","Data":"7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd"} Oct 30 06:20:45.329285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.328791 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" event={"ID":"76c9f39c-2e68-41a5-a7ab-95d3bb40e653","Type":"ContainerDied","Data":"42faa0d7f794f5b0eb550bd4d596d2a5f7c2c95ac332bd66a45184f9f90d239f"} Oct 30 06:20:45.329285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.328820 5980 scope.go:122] "RemoveContainer" containerID="7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd" Oct 30 06:20:45.329285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.328950 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks" Oct 30 06:20:45.342198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.342160 5980 scope.go:122] "RemoveContainer" containerID="7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd" Oct 30 06:20:45.342626 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:45.342588 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd\": container with ID starting with 7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd not found: ID does not exist" containerID="7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd" Oct 30 06:20:45.343590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.343551 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd"} err="failed to get container status \"7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd\": rpc error: code = NotFound desc = could not find container \"7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd\": container with ID starting with 7d505450cda7abebf2ffabd4f18482b43199e2cd2fdb3c145cac02a1969af6dd not found: ID does not exist" Oct 30 06:20:45.389740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.389711 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks"] Oct 30 06:20:45.402565 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.402536 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7ad21-6b469d4b6f-nw2ks"] Oct 30 06:20:45.711816 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.711739 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn"] Oct 30 06:20:45.712504 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.712490 5980 cpu_manager.go:414] "containerMap: removing container" podUID="76c9f39c-2e68-41a5-a7ab-95d3bb40e653" containerName="maas-api" Oct 30 06:20:45.712577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.712507 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c9f39c-2e68-41a5-a7ab-95d3bb40e653" containerName="maas-api" Oct 30 06:20:45.712653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.712644 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c9f39c-2e68-41a5-a7ab-95d3bb40e653" containerName="maas-api" Oct 30 06:20:45.720485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.720452 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.752714 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.752678 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn"] Oct 30 06:20:45.780697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.780664 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.780913 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.780716 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.780913 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.780791 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.780913 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.780813 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.782063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.780940 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.782063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.781127 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.782063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.781162 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8q8s\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.782063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.781228 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.782063 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.781291 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.929689 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.929741 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8q8s\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.929823 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.929851 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.929963 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.930006 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.930092 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.930134 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.930226 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.931030 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.933131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.931308 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.999216 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.985640 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.999216 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.986365 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:45.999216 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:45.987151 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:46.028029 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.007372 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:46.098294 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.098260 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:46.139593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.126090 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:46.139593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.133758 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c9f39c-2e68-41a5-a7ab-95d3bb40e653" path="/var/lib/kubelet/pods/76c9f39c-2e68-41a5-a7ab-95d3bb40e653/volumes" Oct 30 06:20:46.139593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.135172 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8q8s\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:46.363797 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.363768 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:46.515673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.515634 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8"] Oct 30 06:20:46.522606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.522581 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.532459 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.532432 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8"] Oct 30 06:20:46.541654 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.541631 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bf87e1\"/\"openshift-service-ca.crt\"" Oct 30 06:20:46.541654 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.541638 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bf87e1\"/\"kube-root-ca.crt\"" Oct 30 06:20:46.542505 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.542485 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bf87e1\"/\"ipp-route-bf87e1-kserve-self-signed-certs\"" Oct 30 06:20:46.542609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.542590 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bf87e1\"/\"default-dockercfg-8rf72\"" Oct 30 06:20:46.668200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.663908 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjw26\" (UniqueName: \"kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.668200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.664041 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.668200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.664151 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.668200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.664181 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.668200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.664306 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.668200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.664452 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.707137 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:20:46.706165 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45725349_ec16_493f_a1e0_85e3db07452d.slice/crio-ef3453270887395e5b3e46259672448fe2c8b633c7eef52149424375c8f46f65 WatchSource:0}: Error finding container ef3453270887395e5b3e46259672448fe2c8b633c7eef52149424375c8f46f65: Status 404 returned error can't find the container with id ef3453270887395e5b3e46259672448fe2c8b633c7eef52149424375c8f46f65 Oct 30 06:20:46.707137 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.706306 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn"] Oct 30 06:20:46.709778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.709741 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:46.709898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.709808 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:46.709898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.709837 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:46.765987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.765961 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766041 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766069 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766227 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766370 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766399 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766494 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjw26\" (UniqueName: \"kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766664 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.766741 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.766723 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.772701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.769198 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.772701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.769813 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.814570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.813982 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjw26\" (UniqueName: \"kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26\") pod \"ipp-route-bf87e1-kserve-59d894569c-4mdz8\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:46.850720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:46.850693 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:47.296522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.296455 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8"] Oct 30 06:20:47.298582 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:20:47.298543 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48abb063_1fc8_435b_af07_9fd4c43ed944.slice/crio-35c0a9e1c1b9420df4b2d1b7fb1aa97586400610df37b1a0a2ddad6220df023d WatchSource:0}: Error finding container 35c0a9e1c1b9420df4b2d1b7fb1aa97586400610df37b1a0a2ddad6220df023d: Status 404 returned error can't find the container with id 35c0a9e1c1b9420df4b2d1b7fb1aa97586400610df37b1a0a2ddad6220df023d Oct 30 06:20:47.348972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.348237 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" event={"ID":"45725349-ec16-493f-a1e0-85e3db07452d","Type":"ContainerStarted","Data":"55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc"} Oct 30 06:20:47.348972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.348298 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" event={"ID":"45725349-ec16-493f-a1e0-85e3db07452d","Type":"ContainerStarted","Data":"ef3453270887395e5b3e46259672448fe2c8b633c7eef52149424375c8f46f65"} Oct 30 06:20:47.350269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.350241 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" event={"ID":"48abb063-1fc8-435b-af07-9fd4c43ed944","Type":"ContainerStarted","Data":"35c0a9e1c1b9420df4b2d1b7fb1aa97586400610df37b1a0a2ddad6220df023d"} Oct 30 06:20:47.364378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.364355 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:47.365809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.365787 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:47.365917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.365833 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:47.399947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:47.399889 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podStartSLOduration=2.399870184 podStartE2EDuration="2.399870184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:47.397986225 +0000 UTC m=+2901.506025286" watchObservedRunningTime="2026-10-30 06:20:47.399870184 +0000 UTC m=+2901.507909218" Oct 30 06:20:48.359427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.359387 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" event={"ID":"48abb063-1fc8-435b-af07-9fd4c43ed944","Type":"ContainerStarted","Data":"20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0"} Oct 30 06:20:48.372140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.368448 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:48.372140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.368501 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:48.417525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.417474 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" podStartSLOduration=2.41746085 podStartE2EDuration="2.41746085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:48.416245351 +0000 UTC m=+2902.524284383" watchObservedRunningTime="2026-10-30 06:20:48.41746085 +0000 UTC m=+2902.525499874" Oct 30 06:20:48.815287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.815145 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf"] Oct 30 06:20:48.823881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.823178 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.827711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.826821 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf"] Oct 30 06:20:48.852022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.851994 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:48.938837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.938793 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.938837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.938837 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj57r\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.938871 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.938898 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939097 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.938945 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.939204 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.939281 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.939343 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:48.939538 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:48.939519 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041173 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041208 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041248 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041393 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041538 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041468 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041538 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041504 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj57r\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041552 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041590 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041654 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041665 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.041779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041764 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.042869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.041884 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.042869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.042144 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.042869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.042500 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.043825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.043806 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.044276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.044260 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.086178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.079798 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj57r\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.097634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.090566 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.138176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.138146 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:49.402218 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.400678 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:49.402218 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.400736 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:49.405935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.405909 5980 generic.go:356] "Generic (PLEG): container finished" podID="7294cfd8-d116-49bf-9361-a2778bea6118" containerID="3c782c1ce45a2bd66c0c92cd7fe65ea805f1c59448895a6f5ffe5d41ed15bfff" exitCode=0 Oct 30 06:20:49.406091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.405992 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" event={"ID":"7294cfd8-d116-49bf-9361-a2778bea6118","Type":"ContainerDied","Data":"3c782c1ce45a2bd66c0c92cd7fe65ea805f1c59448895a6f5ffe5d41ed15bfff"} Oct 30 06:20:49.406217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.406159 5980 scope.go:122] "RemoveContainer" containerID="f30dc64e597392af6741ec10a007d9797e9130d295b5493f118a88de3d08ebb5" Oct 30 06:20:49.521412 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:20:49.519668 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd231cd83_486c_4e4c_a623_4519bbfa2a37.slice/crio-c7b6673c3069912d12ae4aa2ab0d7864088a88479a094ae2d2b3a8785201f252 WatchSource:0}: Error finding container c7b6673c3069912d12ae4aa2ab0d7864088a88479a094ae2d2b3a8785201f252: Status 404 returned error can't find the container with id c7b6673c3069912d12ae4aa2ab0d7864088a88479a094ae2d2b3a8785201f252 Oct 30 06:20:49.523398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.523358 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:49.523495 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.523452 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:49.523533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.523497 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:49.529640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.529616 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf"] Oct 30 06:20:49.610321 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.609883 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:20:49.653799 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.653727 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.653905 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.653877 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.653976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.653953 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.654035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654000 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:49.654144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654127 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.654206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654189 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.654274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654130 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:49.654326 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654283 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.654392 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654358 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.654449 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654405 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.654510 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.654483 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert\") pod \"7294cfd8-d116-49bf-9361-a2778bea6118\" (UID: \"7294cfd8-d116-49bf-9361-a2778bea6118\") " Oct 30 06:20:49.656688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.656654 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:49.656900 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.656877 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.657161 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.656902 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.657957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.657891 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:49.657957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.657921 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:49.657957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.657924 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:49.659998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.659974 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:49.661492 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.661467 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "kube-api-access-hxflz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:49.672142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.670488 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy" pod "7294cfd8-d116-49bf-9361-a2778bea6118" (UID: "7294cfd8-d116-49bf-9361-a2778bea6118"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:49.718928 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.715834 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn"] Oct 30 06:20:49.730655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.730600 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c"] Oct 30 06:20:49.759466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759428 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxflz\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-kube-api-access-hxflz\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.759466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759462 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.759684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759479 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7294cfd8-d116-49bf-9361-a2778bea6118-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.759684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759491 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7294cfd8-d116-49bf-9361-a2778bea6118-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.759684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759504 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.759684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759516 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7294cfd8-d116-49bf-9361-a2778bea6118-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.759684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:49.759529 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7294cfd8-d116-49bf-9361-a2778bea6118-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:50.386133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.380464 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:50.386133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.380521 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:50.416886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.416854 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" event={"ID":"d231cd83-486c-4e4c-a623-4519bbfa2a37","Type":"ContainerStarted","Data":"9dad03d92625eac23229c29f71b18ea8f8807aaec2fca72eb79aabb6e6669d00"} Oct 30 06:20:50.417402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.416906 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" event={"ID":"d231cd83-486c-4e4c-a623-4519bbfa2a37","Type":"ContainerStarted","Data":"c7b6673c3069912d12ae4aa2ab0d7864088a88479a094ae2d2b3a8785201f252"} Oct 30 06:20:50.418502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.418481 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" Oct 30 06:20:50.418603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.418525 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw" event={"ID":"7294cfd8-d116-49bf-9361-a2778bea6118","Type":"ContainerDied","Data":"a838e330e4350b9fa8b06a630b9cb74958d7ab21f6a4608441e800039b52cc65"} Oct 30 06:20:50.418603 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.418581 5980 scope.go:122] "RemoveContainer" containerID="3c782c1ce45a2bd66c0c92cd7fe65ea805f1c59448895a6f5ffe5d41ed15bfff" Oct 30 06:20:50.501736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.501677 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podStartSLOduration=2.501658485 podStartE2EDuration="2.501658485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:50.49952419 +0000 UTC m=+2904.607563253" watchObservedRunningTime="2026-10-30 06:20:50.501658485 +0000 UTC m=+2904.609697518" Oct 30 06:20:50.543953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.543922 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw"] Oct 30 06:20:50.547312 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:50.547293 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-pj9mw"] Oct 30 06:20:51.139065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:51.139039 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:51.140787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:51.140749 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:20:51.140902 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:51.140807 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:20:51.372517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:51.372480 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:51.372811 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:51.372782 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:52.036040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:52.036008 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" path="/var/lib/kubelet/pods/7294cfd8-d116-49bf-9361-a2778bea6118/volumes" Oct 30 06:20:52.138569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:52.138537 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:20:52.138741 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:52.138582 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:20:52.370703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:52.366408 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:52.370703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:52.366461 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:53.143599 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:53.143566 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:20:53.144281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:53.144226 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:20:53.449720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:53.447936 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:20:53.449720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:53.447989 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:20:53.452281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:53.452084 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t"] Oct 30 06:20:53.452436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:53.452338 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" containerID="cri-o://b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6" gracePeriod=30 Oct 30 06:20:54.138669 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.138637 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:20:54.138953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.138930 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:20:54.378406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.378376 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:54.380534 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.380512 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:54.396405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.396335 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:20:54.752992 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.752895 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7"] Oct 30 06:20:54.753236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.753177 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" containerID="cri-o://8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54" gracePeriod=30 Oct 30 06:20:54.760014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.759983 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd"] Oct 30 06:20:54.760238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:54.760210 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" containerID="cri-o://ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb" gracePeriod=30 Oct 30 06:20:55.019175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:55.019067 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:20:55.019433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:55.019405 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:55.145140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:55.138592 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:20:55.145140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:55.138661 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:20:55.846740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:55.846707 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:20:55.847153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:55.846761 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:20:56.138639 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:56.138547 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:20:56.138639 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:56.138601 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:20:56.859451 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:56.859422 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:56.892000 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:56.891973 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:57.150999 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:57.150913 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:57.151435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:57.151414 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:57.152682 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:57.152662 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:20:57.536405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:57.536330 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:20:58.748207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.748181 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:20:58.893166 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893129 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893202 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893263 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893303 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893458 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893467 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:58.893546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893456 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:58.893546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893530 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893677 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893741 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.893866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.893783 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs\") pod \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\" (UID: \"1fa79cc6-da0a-481f-af58-8777a4ab1e7b\") " Oct 30 06:20:58.894121 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894075 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:58.894321 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894125 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:58.894454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894364 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:58.894919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894898 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.895033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894923 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.895033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894935 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.895033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894947 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.895033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.894961 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.896009 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.895782 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:58.896570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.896224 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "kube-api-access-pzb96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:58.896570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.896277 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:58.897324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.897301 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo" pod "1fa79cc6-da0a-481f-af58-8777a4ab1e7b" (UID: "1fa79cc6-da0a-481f-af58-8777a4ab1e7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:58.996769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.996670 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.996769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.996703 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.996769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.996715 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.996769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:58.996726 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzb96\" (UniqueName: \"kubernetes.io/projected/1fa79cc6-da0a-481f-af58-8777a4ab1e7b-kube-api-access-pzb96\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.545640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.545602 5980 generic.go:356] "Generic (PLEG): container finished" podID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerID="b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6" exitCode=0 Oct 30 06:20:59.545841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.545793 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" event={"ID":"1fa79cc6-da0a-481f-af58-8777a4ab1e7b","Type":"ContainerDied","Data":"b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6"} Oct 30 06:20:59.545932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.545891 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" event={"ID":"1fa79cc6-da0a-481f-af58-8777a4ab1e7b","Type":"ContainerDied","Data":"97b1ef9df759f50a3db2927463891ad859cef7f2f4049ea237e629f11eeb3076"} Oct 30 06:20:59.545995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.545937 5980 scope.go:122] "RemoveContainer" containerID="b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6" Oct 30 06:20:59.546327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.546177 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t" Oct 30 06:20:59.562405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.562381 5980 scope.go:122] "RemoveContainer" containerID="a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3" Oct 30 06:20:59.573546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.573332 5980 scope.go:122] "RemoveContainer" containerID="b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6" Oct 30 06:20:59.573631 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:59.573611 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6\": container with ID starting with b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6 not found: ID does not exist" containerID="b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6" Oct 30 06:20:59.573705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.573639 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6"} err="failed to get container status \"b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6\": rpc error: code = NotFound desc = could not find container \"b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6\": container with ID starting with b007e3cc60abdd9c1697d6c0180c4ea6b0bd45fd5aab202d938eb1b798ae63c6 not found: ID does not exist" Oct 30 06:20:59.573705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.573657 5980 scope.go:122] "RemoveContainer" containerID="a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3" Oct 30 06:20:59.573888 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:20:59.573871 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3\": container with ID starting with a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3 not found: ID does not exist" containerID="a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3" Oct 30 06:20:59.573944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.573890 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3"} err="failed to get container status \"a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3\": rpc error: code = NotFound desc = could not find container \"a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3\": container with ID starting with a293745a224dfbdfac9c55667062b26e0a3e9782fe9f06d9057ae936220e64a3 not found: ID does not exist" Oct 30 06:20:59.627174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.627144 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t"] Oct 30 06:20:59.630639 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:20:59.630614 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f7ad21-openshift-default-67bb74ff5f-n6h5t"] Oct 30 06:21:00.024402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.024372 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" path="/var/lib/kubelet/pods/1fa79cc6-da0a-481f-af58-8777a4ab1e7b/volumes" Oct 30 06:21:00.062628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.062273 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:21:00.141660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.140664 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:21:00.215692 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.215663 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.215887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.215724 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.215887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.215777 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.215887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.215817 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.215887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.215860 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.215949 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216010 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216069 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.216133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216087 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216137 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216163 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216284 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216232 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216290 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216413 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216371 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.216413 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216402 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216492 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216536 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216572 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216560 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216590 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert\") pod \"268a9a7a-53a1-4d01-922f-4b444ba492b6\" (UID: \"268a9a7a-53a1-4d01-922f-4b444ba492b6\") " Oct 30 06:21:00.216670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216647 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data\") pod \"d87a41c6-002e-417e-a0aa-0e075df529e7\" (UID: \"d87a41c6-002e-417e-a0aa-0e075df529e7\") " Oct 30 06:21:00.216799 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216671 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.216799 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216689 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.216907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216799 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:00.216968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.216925 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.217128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217087 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.217363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217300 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.217870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217847 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217861 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217875 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217906 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217922 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d87a41c6-002e-417e-a0aa-0e075df529e7-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217935 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217947 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.217958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217959 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.218405 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.217973 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.218867 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.218821 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.220897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.220864 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:00.221379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.221282 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.221379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.221313 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "kube-api-access-4t96c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.221379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.221353 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.221612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.221540 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:00.221802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.221779 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.222323 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.221959 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh" pod "d87a41c6-002e-417e-a0aa-0e075df529e7" (UID: "d87a41c6-002e-417e-a0aa-0e075df529e7"). InnerVolumeSpecName "kube-api-access-gg4qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.222664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.222646 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token" pod "268a9a7a-53a1-4d01-922f-4b444ba492b6" (UID: "268a9a7a-53a1-4d01-922f-4b444ba492b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.321562 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321527 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321567 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321581 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321594 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/268a9a7a-53a1-4d01-922f-4b444ba492b6-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321606 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321619 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg4qh\" (UniqueName: \"kubernetes.io/projected/d87a41c6-002e-417e-a0aa-0e075df529e7-kube-api-access-gg4qh\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321632 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d87a41c6-002e-417e-a0aa-0e075df529e7-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321646 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t96c\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-kube-api-access-4t96c\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321659 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.321757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.321673 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/268a9a7a-53a1-4d01-922f-4b444ba492b6-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.555620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.555589 5980 generic.go:356] "Generic (PLEG): container finished" podID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerID="ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb" exitCode=0 Oct 30 06:21:00.555791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.555631 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" Oct 30 06:21:00.555791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.555670 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" event={"ID":"268a9a7a-53a1-4d01-922f-4b444ba492b6","Type":"ContainerDied","Data":"ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb"} Oct 30 06:21:00.555791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.555722 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd" event={"ID":"268a9a7a-53a1-4d01-922f-4b444ba492b6","Type":"ContainerDied","Data":"956da2c1bd10e286fe54ced9f606b75f936ae4ee02bbc8dcdb79083cf31923be"} Oct 30 06:21:00.555791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.555753 5980 scope.go:122] "RemoveContainer" containerID="ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb" Oct 30 06:21:00.557306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.557283 5980 generic.go:356] "Generic (PLEG): container finished" podID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerID="8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54" exitCode=0 Oct 30 06:21:00.557590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.557317 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" event={"ID":"d87a41c6-002e-417e-a0aa-0e075df529e7","Type":"ContainerDied","Data":"8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54"} Oct 30 06:21:00.557590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.557338 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" Oct 30 06:21:00.557590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.557355 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7" event={"ID":"d87a41c6-002e-417e-a0aa-0e075df529e7","Type":"ContainerDied","Data":"24bfca9b0be65e771f6d63d2299d43538b549a352cea8a9a1d91f23ad14ee980"} Oct 30 06:21:00.575478 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.574815 5980 scope.go:122] "RemoveContainer" containerID="ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb" Oct 30 06:21:00.576436 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:00.575624 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb\": container with ID starting with ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb not found: ID does not exist" containerID="ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb" Oct 30 06:21:00.576436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.575651 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb"} err="failed to get container status \"ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb\": rpc error: code = NotFound desc = could not find container \"ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb\": container with ID starting with ee18075e76de3466e6b1c429348b0f30ddec38be12a01307ff5d25d351b801fb not found: ID does not exist" Oct 30 06:21:00.576436 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.575668 5980 scope.go:122] "RemoveContainer" containerID="8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54" Oct 30 06:21:00.588800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.588781 5980 scope.go:122] "RemoveContainer" containerID="8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54" Oct 30 06:21:00.589488 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:00.589462 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54\": container with ID starting with 8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54 not found: ID does not exist" containerID="8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54" Oct 30 06:21:00.589578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.589498 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54"} err="failed to get container status \"8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54\": rpc error: code = NotFound desc = could not find container \"8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54\": container with ID starting with 8bd9401d2e0c88787a4b7358e3a01516389f6dba5a422cca35e58a5cee089d54 not found: ID does not exist" Oct 30 06:21:00.617087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.617064 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7"] Oct 30 06:21:00.628893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.628867 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-gbcd7"] Oct 30 06:21:00.657814 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.657746 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd"] Oct 30 06:21:00.661695 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:00.661671 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-8rxzd"] Oct 30 06:21:02.052770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:02.052726 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" path="/var/lib/kubelet/pods/268a9a7a-53a1-4d01-922f-4b444ba492b6/volumes" Oct 30 06:21:02.053626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:02.053604 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" path="/var/lib/kubelet/pods/d87a41c6-002e-417e-a0aa-0e075df529e7/volumes" Oct 30 06:21:04.454043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:04.454009 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf"] Oct 30 06:21:04.487577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:04.487550 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr"] Oct 30 06:21:04.774632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:04.774552 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" containerID="cri-o://3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb" gracePeriod=45 Oct 30 06:21:04.796722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:04.796689 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" containerID="cri-o://cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b" gracePeriod=45 Oct 30 06:21:05.146162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.146096 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 06:21:05.193167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.190702 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.426706 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache\") pod \"249063da-915b-4eaa-a85f-643d37c75157\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.426815 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm\") pod \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.426954 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl\" (UniqueName: \"kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl\") pod \"249063da-915b-4eaa-a85f-643d37c75157\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.426999 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir\") pod \"249063da-915b-4eaa-a85f-643d37c75157\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.427057 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home\") pod \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.427146 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs\") pod \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.427189 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache\") pod \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.427222 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs\" (UniqueName: \"kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs\") pod \"249063da-915b-4eaa-a85f-643d37c75157\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.427268 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4\" (UniqueName: \"kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4\") pod \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " Oct 30 06:21:05.429500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.427311 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home\") pod \"249063da-915b-4eaa-a85f-643d37c75157\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " Oct 30 06:21:05.431374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.430539 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir\") pod \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\" (UID: \"9f906dd9-e8c3-4be2-9079-3b1aa10970f9\") " Oct 30 06:21:05.431374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.430666 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm\") pod \"249063da-915b-4eaa-a85f-643d37c75157\" (UID: \"249063da-915b-4eaa-a85f-643d37c75157\") " Oct 30 06:21:05.433745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.433464 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir" pod "249063da-915b-4eaa-a85f-643d37c75157" (UID: "249063da-915b-4eaa-a85f-643d37c75157"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.435174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.434832 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir" pod "9f906dd9-e8c3-4be2-9079-3b1aa10970f9" (UID: "9f906dd9-e8c3-4be2-9079-3b1aa10970f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.435174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.435011 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home" pod "249063da-915b-4eaa-a85f-643d37c75157" (UID: "249063da-915b-4eaa-a85f-643d37c75157"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.435851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.435351 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache" pod "9f906dd9-e8c3-4be2-9079-3b1aa10970f9" (UID: "9f906dd9-e8c3-4be2-9079-3b1aa10970f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.435851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.435628 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache" pod "249063da-915b-4eaa-a85f-643d37c75157" (UID: "249063da-915b-4eaa-a85f-643d37c75157"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.436243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.436224 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home" pod "9f906dd9-e8c3-4be2-9079-3b1aa10970f9" (UID: "9f906dd9-e8c3-4be2-9079-3b1aa10970f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.439175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.437138 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs" pod "9f906dd9-e8c3-4be2-9079-3b1aa10970f9" (UID: "9f906dd9-e8c3-4be2-9079-3b1aa10970f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:05.479720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.479266 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl" pod "249063da-915b-4eaa-a85f-643d37c75157" (UID: "249063da-915b-4eaa-a85f-643d37c75157"). InnerVolumeSpecName "kube-api-access-bnvcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:05.479720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.479398 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4" pod "9f906dd9-e8c3-4be2-9079-3b1aa10970f9" (UID: "9f906dd9-e8c3-4be2-9079-3b1aa10970f9"). InnerVolumeSpecName "kube-api-access-w9nw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:05.479720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.479488 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm" pod "249063da-915b-4eaa-a85f-643d37c75157" (UID: "249063da-915b-4eaa-a85f-643d37c75157"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.487199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.487172 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs" pod "249063da-915b-4eaa-a85f-643d37c75157" (UID: "249063da-915b-4eaa-a85f-643d37c75157"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:05.487199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.487180 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm" pod "9f906dd9-e8c3-4be2-9079-3b1aa10970f9" (UID: "9f906dd9-e8c3-4be2-9079-3b1aa10970f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:05.533362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533327 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvcl\" (UniqueName: \"kubernetes.io/projected/249063da-915b-4eaa-a85f-643d37c75157-kube-api-access-bnvcl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533353 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533362 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533363 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533371 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533382 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533390 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/249063da-915b-4eaa-a85f-643d37c75157-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533398 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nw4\" (UniqueName: \"kubernetes.io/projected/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-kube-api-access-w9nw4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533405 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533413 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533420 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533427 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/249063da-915b-4eaa-a85f-643d37c75157-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.533651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.533434 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f906dd9-e8c3-4be2-9079-3b1aa10970f9-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:05.584144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.584076 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx"] Oct 30 06:21:05.584378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.584352 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" podUID="4896357c-db54-4af4-9c1c-d08559326b79" containerName="maas-api" containerID="cri-o://f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7" gracePeriod=30 Oct 30 06:21:05.627929 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.627894 5980 generic.go:356] "Generic (PLEG): container finished" podID="249063da-915b-4eaa-a85f-643d37c75157" containerID="cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b" exitCode=0 Oct 30 06:21:05.628093 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.627987 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" event={"ID":"249063da-915b-4eaa-a85f-643d37c75157","Type":"ContainerDied","Data":"cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b"} Oct 30 06:21:05.628093 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.628023 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" event={"ID":"249063da-915b-4eaa-a85f-643d37c75157","Type":"ContainerDied","Data":"57eeee0f3a31839522876b76d7720a839988edeabf413187c989c58be8e9e533"} Oct 30 06:21:05.628093 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.628044 5980 scope.go:122] "RemoveContainer" containerID="cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b" Oct 30 06:21:05.628333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.628246 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn" Oct 30 06:21:05.640004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.639961 5980 generic.go:356] "Generic (PLEG): container finished" podID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerID="3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb" exitCode=0 Oct 30 06:21:05.640208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.640092 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" event={"ID":"9f906dd9-e8c3-4be2-9079-3b1aa10970f9","Type":"ContainerDied","Data":"3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb"} Oct 30 06:21:05.640208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.640161 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" event={"ID":"9f906dd9-e8c3-4be2-9079-3b1aa10970f9","Type":"ContainerDied","Data":"58c4d179b5cea17abe6a9f6c0afba8f0702c025e18b726fbc53d97988e9edf66"} Oct 30 06:21:05.640408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.640220 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c" Oct 30 06:21:05.706158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.706033 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c"] Oct 30 06:21:05.709978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.709945 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/premium-w0-197e9b-kserve-79984d9bcc-hwt6c"] Oct 30 06:21:05.711010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.710926 5980 scope.go:122] "RemoveContainer" containerID="cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b" Oct 30 06:21:05.711408 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:05.711383 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b\": container with ID starting with cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b not found: ID does not exist" containerID="cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b" Oct 30 06:21:05.712769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.711414 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b"} err="failed to get container status \"cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b\": rpc error: code = NotFound desc = could not find container \"cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b\": container with ID starting with cd27ee3c8a80c719ea4e5f3c00411996379aef6b6c6aa613bd06d1a33482d49b not found: ID does not exist" Oct 30 06:21:05.712769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.711441 5980 scope.go:122] "RemoveContainer" containerID="3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb" Oct 30 06:21:05.724752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.724723 5980 scope.go:122] "RemoveContainer" containerID="3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb" Oct 30 06:21:05.725093 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:05.725056 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb\": container with ID starting with 3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb not found: ID does not exist" containerID="3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb" Oct 30 06:21:05.725211 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.725100 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb"} err="failed to get container status \"3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb\": rpc error: code = NotFound desc = could not find container \"3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb\": container with ID starting with 3fe5470cf422c6ceee7471e686e3eda0fe23f26feffad2d2845b66b41c1cc1eb not found: ID does not exist" Oct 30 06:21:05.753992 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.753937 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn"] Oct 30 06:21:05.760309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.760277 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-197e9b/distinct-w0-197e9b-kserve-756cfdb47b-qhfpn"] Oct 30 06:21:05.881188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.881153 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9204-d5ade7"] Oct 30 06:21:05.882240 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882218 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882241 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882255 5980 cpu_manager.go:414] "containerMap: removing container" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882263 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882281 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882288 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882302 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882309 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882328 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882335 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" Oct 30 06:21:05.882348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882349 5980 cpu_manager.go:414] "containerMap: removing container" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882356 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882379 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882387 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882589 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" containerName="main" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882600 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="249063da-915b-4eaa-a85f-643d37c75157" containerName="main" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882611 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882621 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882632 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87a41c6-002e-417e-a0aa-0e075df529e7" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882642 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="268a9a7a-53a1-4d01-922f-4b444ba492b6" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882653 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa79cc6-da0a-481f-af58-8777a4ab1e7b" containerName="istio-proxy" Oct 30 06:21:05.882859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.882665 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" Oct 30 06:21:05.886905 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.886820 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:05.896168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.896142 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9204-d5ade7"] Oct 30 06:21:05.906482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.906457 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg"] Oct 30 06:21:05.906771 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.906748 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4m56v\"" Oct 30 06:21:05.942968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.942939 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8"] Oct 30 06:21:05.961767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:05.961695 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:21:06.024144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.023034 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249063da-915b-4eaa-a85f-643d37c75157" path="/var/lib/kubelet/pods/249063da-915b-4eaa-a85f-643d37c75157/volumes" Oct 30 06:21:06.024144 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.023813 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f906dd9-e8c3-4be2-9079-3b1aa10970f9" path="/var/lib/kubelet/pods/9f906dd9-e8c3-4be2-9079-3b1aa10970f9/volumes" Oct 30 06:21:06.052889 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.050948 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljnq\" (UniqueName: \"kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq\") pod \"test-curl-9204-d5ade7\" (UID: \"88af9da4-0208-4014-84ba-d86f3ed1e7d0\") " pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:06.152491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.152446 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs\") pod \"4896357c-db54-4af4-9c1c-d08559326b79\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " Oct 30 06:21:06.152901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.152693 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca\") pod \"4896357c-db54-4af4-9c1c-d08559326b79\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " Oct 30 06:21:06.152901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.152759 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls\") pod \"4896357c-db54-4af4-9c1c-d08559326b79\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " Oct 30 06:21:06.153042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.152912 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz\" (UniqueName: \"kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz\") pod \"4896357c-db54-4af4-9c1c-d08559326b79\" (UID: \"4896357c-db54-4af4-9c1c-d08559326b79\") " Oct 30 06:21:06.153708 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.153683 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljnq\" (UniqueName: \"kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq\") pod \"test-curl-9204-d5ade7\" (UID: \"88af9da4-0208-4014-84ba-d86f3ed1e7d0\") " pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:06.155672 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.155648 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca" pod "4896357c-db54-4af4-9c1c-d08559326b79" (UID: "4896357c-db54-4af4-9c1c-d08559326b79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:06.162182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.162089 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls" pod "4896357c-db54-4af4-9c1c-d08559326b79" (UID: "4896357c-db54-4af4-9c1c-d08559326b79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:06.162470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.162436 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz" pod "4896357c-db54-4af4-9c1c-d08559326b79" (UID: "4896357c-db54-4af4-9c1c-d08559326b79"). InnerVolumeSpecName "kube-api-access-bf9xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:06.211160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.190232 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs" pod "4896357c-db54-4af4-9c1c-d08559326b79" (UID: "4896357c-db54-4af4-9c1c-d08559326b79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:06.211160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.192063 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljnq\" (UniqueName: \"kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq\") pod \"test-curl-9204-d5ade7\" (UID: \"88af9da4-0208-4014-84ba-d86f3ed1e7d0\") " pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:06.211160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.210433 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:06.288160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.265559 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf9xz\" (UniqueName: \"kubernetes.io/projected/4896357c-db54-4af4-9c1c-d08559326b79-kube-api-access-bf9xz\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.288160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.265594 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.288160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.265611 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4896357c-db54-4af4-9c1c-d08559326b79-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.288160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.265627 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4896357c-db54-4af4-9c1c-d08559326b79-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.654757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.653817 5980 generic.go:356] "Generic (PLEG): container finished" podID="4896357c-db54-4af4-9c1c-d08559326b79" containerID="f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7" exitCode=0 Oct 30 06:21:06.655283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.654892 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" event={"ID":"4896357c-db54-4af4-9c1c-d08559326b79","Type":"ContainerDied","Data":"f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7"} Oct 30 06:21:06.655283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.654976 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" event={"ID":"4896357c-db54-4af4-9c1c-d08559326b79","Type":"ContainerDied","Data":"b721b8ce649d63817d0862442a26e34de0690dff411a7e9181fa5cdad0b0ad4f"} Oct 30 06:21:06.655283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.655007 5980 scope.go:122] "RemoveContainer" containerID="f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7" Oct 30 06:21:06.655283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.655175 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx" Oct 30 06:21:06.681876 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.681853 5980 scope.go:122] "RemoveContainer" containerID="f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7" Oct 30 06:21:06.682296 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:06.682259 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7\": container with ID starting with f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7 not found: ID does not exist" containerID="f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7" Oct 30 06:21:06.682416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.682300 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7"} err="failed to get container status \"f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7\": rpc error: code = NotFound desc = could not find container \"f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7\": container with ID starting with f66084651ae11efa08b6828d5be8499ef046aded04ef4172b55409e25e2b05c7 not found: ID does not exist" Oct 30 06:21:06.717881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.717847 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9204-d5ade7"] Oct 30 06:21:06.718507 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:21:06.718481 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88af9da4_0208_4014_84ba_d86f3ed1e7d0.slice/crio-68b2274803f2687a57433f8c003a1aeb3eed762ccd9c539d914ea1e4a2657e11 WatchSource:0}: Error finding container 68b2274803f2687a57433f8c003a1aeb3eed762ccd9c539d914ea1e4a2657e11: Status 404 returned error can't find the container with id 68b2274803f2687a57433f8c003a1aeb3eed762ccd9c539d914ea1e4a2657e11 Oct 30 06:21:06.721286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.721267 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:21:06.731420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.731396 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx"] Oct 30 06:21:06.739511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:06.739488 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dfc3d2-74884884cf-47ldx"] Oct 30 06:21:07.672059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:07.672020 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9204-d5ade7" event={"ID":"88af9da4-0208-4014-84ba-d86f3ed1e7d0","Type":"ContainerStarted","Data":"68b2274803f2687a57433f8c003a1aeb3eed762ccd9c539d914ea1e4a2657e11"} Oct 30 06:21:08.015868 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:08.015800 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4896357c-db54-4af4-9c1c-d08559326b79" path="/var/lib/kubelet/pods/4896357c-db54-4af4-9c1c-d08559326b79/volumes" Oct 30 06:21:09.455132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.455081 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" podUID="cc0e8018-070f-4add-ab40-569613d1effa" containerName="payload-processing" containerID="cri-o://58d804fdd42dc9f7d874d4d0b952c467a957470d1841f2262e1459d0f663cdf4" gracePeriod=25 Oct 30 06:21:09.488188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.488093 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" podUID="07a48d1d-d63f-4a65-856d-2487f232727f" containerName="payload-pre-processing" containerID="cri-o://65c7a41c631abc1423bfbda33def343f258ede724881770eefe056f03bf2af20" gracePeriod=25 Oct 30 06:21:09.713997 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.713889 5980 generic.go:356] "Generic (PLEG): container finished" podID="cc0e8018-070f-4add-ab40-569613d1effa" containerID="58d804fdd42dc9f7d874d4d0b952c467a957470d1841f2262e1459d0f663cdf4" exitCode=0 Oct 30 06:21:09.713997 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.713940 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" event={"ID":"cc0e8018-070f-4add-ab40-569613d1effa","Type":"ContainerDied","Data":"58d804fdd42dc9f7d874d4d0b952c467a957470d1841f2262e1459d0f663cdf4"} Oct 30 06:21:09.715831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.715810 5980 generic.go:356] "Generic (PLEG): container finished" podID="07a48d1d-d63f-4a65-856d-2487f232727f" containerID="65c7a41c631abc1423bfbda33def343f258ede724881770eefe056f03bf2af20" exitCode=0 Oct 30 06:21:09.715990 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.715979 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" event={"ID":"07a48d1d-d63f-4a65-856d-2487f232727f","Type":"ContainerDied","Data":"65c7a41c631abc1423bfbda33def343f258ede724881770eefe056f03bf2af20"} Oct 30 06:21:09.794347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.794312 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 06:21:09.830423 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.830402 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 06:21:09.895897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.895864 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume\") pod \"cc0e8018-070f-4add-ab40-569613d1effa\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " Oct 30 06:21:09.896569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.895920 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume\") pod \"07a48d1d-d63f-4a65-856d-2487f232727f\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " Oct 30 06:21:09.896569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.895985 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl\" (UniqueName: \"kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl\") pod \"cc0e8018-070f-4add-ab40-569613d1effa\" (UID: \"cc0e8018-070f-4add-ab40-569613d1effa\") " Oct 30 06:21:09.896569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.896058 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625\" (UniqueName: \"kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625\") pod \"07a48d1d-d63f-4a65-856d-2487f232727f\" (UID: \"07a48d1d-d63f-4a65-856d-2487f232727f\") " Oct 30 06:21:09.897612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.897582 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume" pod "07a48d1d-d63f-4a65-856d-2487f232727f" (UID: "07a48d1d-d63f-4a65-856d-2487f232727f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:09.897735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.897661 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume" pod "cc0e8018-070f-4add-ab40-569613d1effa" (UID: "cc0e8018-070f-4add-ab40-569613d1effa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:09.899853 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.899823 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625" pod "07a48d1d-d63f-4a65-856d-2487f232727f" (UID: "07a48d1d-d63f-4a65-856d-2487f232727f"). InnerVolumeSpecName "kube-api-access-x5625". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:09.900745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.900712 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl" pod "cc0e8018-070f-4add-ab40-569613d1effa" (UID: "cc0e8018-070f-4add-ab40-569613d1effa"). InnerVolumeSpecName "kube-api-access-4f8dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:09.998440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.998361 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e8018-070f-4add-ab40-569613d1effa-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.998440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.998385 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07a48d1d-d63f-4a65-856d-2487f232727f-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.998440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.998395 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8dl\" (UniqueName: \"kubernetes.io/projected/cc0e8018-070f-4add-ab40-569613d1effa-kube-api-access-4f8dl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.998440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:09.998403 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5625\" (UniqueName: \"kubernetes.io/projected/07a48d1d-d63f-4a65-856d-2487f232727f-kube-api-access-x5625\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:10.724389 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.724345 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" event={"ID":"cc0e8018-070f-4add-ab40-569613d1effa","Type":"ContainerDied","Data":"56e00851420e612dd32892f7d44edbb8b8de196f7baa920c6619527f25442cde"} Oct 30 06:21:10.724865 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.724363 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf" Oct 30 06:21:10.724865 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.724430 5980 scope.go:122] "RemoveContainer" containerID="58d804fdd42dc9f7d874d4d0b952c467a957470d1841f2262e1459d0f663cdf4" Oct 30 06:21:10.725840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.725819 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" Oct 30 06:21:10.725964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.725846 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr" event={"ID":"07a48d1d-d63f-4a65-856d-2487f232727f","Type":"ContainerDied","Data":"fa8f5e2c6eecfaa88a8b6a71abccb5ac08365a2b424a5eaa859674dd2c1a796a"} Oct 30 06:21:10.735209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.735195 5980 scope.go:122] "RemoveContainer" containerID="65c7a41c631abc1423bfbda33def343f258ede724881770eefe056f03bf2af20" Oct 30 06:21:10.783363 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.783335 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf"] Oct 30 06:21:10.797179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.797152 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-477f6b-58f667c489-blxxf"] Oct 30 06:21:10.828092 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.828064 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr"] Oct 30 06:21:10.835947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.835927 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-477f6b-7b75ffd5f5-fzcnr"] Oct 30 06:21:10.907630 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.907590 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerName="payload-processing" containerID="cri-o://9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017" gracePeriod=25 Oct 30 06:21:10.943327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:10.943291 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerName="payload-pre-processing" containerID="cri-o://b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb" gracePeriod=25 Oct 30 06:21:12.018793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:12.018759 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a48d1d-d63f-4a65-856d-2487f232727f" path="/var/lib/kubelet/pods/07a48d1d-d63f-4a65-856d-2487f232727f/volumes" Oct 30 06:21:12.019467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:12.019445 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0e8018-070f-4add-ab40-569613d1effa" path="/var/lib/kubelet/pods/cc0e8018-070f-4add-ab40-569613d1effa/volumes" Oct 30 06:21:13.091898 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:13.091863 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:21:15.447924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:15.447888 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs"] Oct 30 06:21:16.007197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:16.007155 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:16.007503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:16.007468 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:20.817802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:20.817766 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9204-d5ade7" event={"ID":"88af9da4-0208-4014-84ba-d86f3ed1e7d0","Type":"ContainerStarted","Data":"1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d"} Oct 30 06:21:20.859636 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:20.859591 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9204-d5ade7" podStartSLOduration=2.3090077 podStartE2EDuration="15.859578584s" totalImagesPullingTime="13.550570884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:20.858617268 +0000 UTC m=+2934.966656302" watchObservedRunningTime="2026-10-30 06:21:20.859578584 +0000 UTC m=+2934.967617617" Oct 30 06:21:21.798186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:21.798148 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9204-d5ade7"] Oct 30 06:21:21.802736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:21.802703 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9204-d5ade7"] Oct 30 06:21:22.835504 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:22.835463 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9204-d5ade7" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" containerName="test-curl-9204-d5ade7" containerID="cri-o://1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d" gracePeriod=2 Oct 30 06:21:25.083972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.083944 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:25.154399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.154330 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq\" (UniqueName: \"kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq\") pod \"88af9da4-0208-4014-84ba-d86f3ed1e7d0\" (UID: \"88af9da4-0208-4014-84ba-d86f3ed1e7d0\") " Oct 30 06:21:25.156637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.156613 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq" pod "88af9da4-0208-4014-84ba-d86f3ed1e7d0" (UID: "88af9da4-0208-4014-84ba-d86f3ed1e7d0"). InnerVolumeSpecName "kube-api-access-zljnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:25.256934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.256903 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljnq\" (UniqueName: \"kubernetes.io/projected/88af9da4-0208-4014-84ba-d86f3ed1e7d0-kube-api-access-zljnq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:25.858070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.858034 5980 generic.go:356] "Generic (PLEG): container finished" podID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" containerID="1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d" exitCode=137 Oct 30 06:21:25.858276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.858093 5980 scope.go:122] "RemoveContainer" containerID="1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d" Oct 30 06:21:25.858276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.858144 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:25.870701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.870658 5980 scope.go:122] "RemoveContainer" containerID="1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d" Oct 30 06:21:25.871181 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:25.871145 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d\": container with ID starting with 1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d not found: ID does not exist" containerID="1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d" Oct 30 06:21:25.871341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:25.871311 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d"} err="failed to get container status \"1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d\": rpc error: code = NotFound desc = could not find container \"1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d\": container with ID starting with 1983fa3e1e1b10ede7065afa2dbe2e4febd30306a10bc4fed0afc92ecca9864d not found: ID does not exist" Oct 30 06:21:25.886323 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:25.886296 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9204-d5ade7\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:26.007475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:26.007435 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:26.007669 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:26.007516 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:26.031378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:26.031346 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" path="/var/lib/kubelet/pods/88af9da4-0208-4014-84ba-d86f3ed1e7d0/volumes" Oct 30 06:21:26.032289 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:26.032262 5980 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9204-d5ade7\" is forbidden: User \"system:node:ip-10-0-138-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-15.ec2.internal' and this object" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" pod="opendatahub/test-curl-9204-d5ade7" Oct 30 06:21:30.471398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.471351 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" podUID="5052f3d9-b903-410f-83ba-7b0cd797e710" containerName="main" containerID="cri-o://a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d" gracePeriod=45 Oct 30 06:21:30.782524 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.782497 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:21:30.934391 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.934356 5980 generic.go:356] "Generic (PLEG): container finished" podID="5052f3d9-b903-410f-83ba-7b0cd797e710" containerID="a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d" exitCode=0 Oct 30 06:21:30.934517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.934402 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" Oct 30 06:21:30.934517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.934433 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" event={"ID":"5052f3d9-b903-410f-83ba-7b0cd797e710","Type":"ContainerDied","Data":"a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d"} Oct 30 06:21:30.934517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.934483 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs" event={"ID":"5052f3d9-b903-410f-83ba-7b0cd797e710","Type":"ContainerDied","Data":"9ae360727d9cf62b8a31f25abeae925c8ff1564e9a99cff8fca273419d98453f"} Oct 30 06:21:30.934517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.934506 5980 scope.go:122] "RemoveContainer" containerID="a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d" Oct 30 06:21:30.953901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.953880 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm\") pod \"5052f3d9-b903-410f-83ba-7b0cd797e710\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " Oct 30 06:21:30.954024 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.953928 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home\") pod \"5052f3d9-b903-410f-83ba-7b0cd797e710\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " Oct 30 06:21:30.954158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954094 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir\") pod \"5052f3d9-b903-410f-83ba-7b0cd797e710\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " Oct 30 06:21:30.954272 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954248 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home" pod "5052f3d9-b903-410f-83ba-7b0cd797e710" (UID: "5052f3d9-b903-410f-83ba-7b0cd797e710"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:30.954344 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954311 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n\" (UniqueName: \"kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n\") pod \"5052f3d9-b903-410f-83ba-7b0cd797e710\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " Oct 30 06:21:30.954344 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954329 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir" pod "5052f3d9-b903-410f-83ba-7b0cd797e710" (UID: "5052f3d9-b903-410f-83ba-7b0cd797e710"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:30.954477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954360 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache\") pod \"5052f3d9-b903-410f-83ba-7b0cd797e710\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " Oct 30 06:21:30.954477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954444 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs\" (UniqueName: \"kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs\") pod \"5052f3d9-b903-410f-83ba-7b0cd797e710\" (UID: \"5052f3d9-b903-410f-83ba-7b0cd797e710\") " Oct 30 06:21:30.955153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.954641 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache" pod "5052f3d9-b903-410f-83ba-7b0cd797e710" (UID: "5052f3d9-b903-410f-83ba-7b0cd797e710"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:30.955153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.955064 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:30.955153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.955078 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:30.955153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.955102 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:30.956446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.956419 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm" pod "5052f3d9-b903-410f-83ba-7b0cd797e710" (UID: "5052f3d9-b903-410f-83ba-7b0cd797e710"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:30.956797 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.956776 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs" pod "5052f3d9-b903-410f-83ba-7b0cd797e710" (UID: "5052f3d9-b903-410f-83ba-7b0cd797e710"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:30.957268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:30.957243 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n" pod "5052f3d9-b903-410f-83ba-7b0cd797e710" (UID: "5052f3d9-b903-410f-83ba-7b0cd797e710"). InnerVolumeSpecName "kube-api-access-dd46n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:31.056691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.056660 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd46n\" (UniqueName: \"kubernetes.io/projected/5052f3d9-b903-410f-83ba-7b0cd797e710-kube-api-access-dd46n\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.056825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.056692 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5052f3d9-b903-410f-83ba-7b0cd797e710-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.056825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.056708 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5052f3d9-b903-410f-83ba-7b0cd797e710-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.057670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.057648 5980 scope.go:122] "RemoveContainer" containerID="a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d" Oct 30 06:21:31.058231 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:31.058211 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d\": container with ID starting with a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d not found: ID does not exist" containerID="a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d" Oct 30 06:21:31.058324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.058240 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d"} err="failed to get container status \"a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d\": rpc error: code = NotFound desc = could not find container \"a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d\": container with ID starting with a28a6e8bd1563b00bcc7582f7ad7067d4c30901ab7e25a7ba9d755abe6a4610d not found: ID does not exist" Oct 30 06:21:31.222727 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.222703 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:21:31.274967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.274942 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:21:31.311617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.311539 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs"] Oct 30 06:21:31.316525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.316495 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/rate-test-model-24db22-kserve-7b8c9944d9-gtgvs"] Oct 30 06:21:31.361426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.361393 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume\") pod \"3699cf25-4f57-42c8-b114-ff2fb561d189\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " Oct 30 06:21:31.361586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.361445 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx\" (UniqueName: \"kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx\") pod \"3699cf25-4f57-42c8-b114-ff2fb561d189\" (UID: \"3699cf25-4f57-42c8-b114-ff2fb561d189\") " Oct 30 06:21:31.361723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.361664 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume\") pod \"dfa83e29-78fd-457d-911f-1a828ec86b4e\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " Oct 30 06:21:31.361877 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.361823 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume" pod "3699cf25-4f57-42c8-b114-ff2fb561d189" (UID: "3699cf25-4f57-42c8-b114-ff2fb561d189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:31.362461 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.361890 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc\" (UniqueName: \"kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc\") pod \"dfa83e29-78fd-457d-911f-1a828ec86b4e\" (UID: \"dfa83e29-78fd-457d-911f-1a828ec86b4e\") " Oct 30 06:21:31.362461 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.362291 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume" pod "dfa83e29-78fd-457d-911f-1a828ec86b4e" (UID: "dfa83e29-78fd-457d-911f-1a828ec86b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:31.362655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.362632 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3699cf25-4f57-42c8-b114-ff2fb561d189-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.362811 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.362796 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa83e29-78fd-457d-911f-1a828ec86b4e-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.364377 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.364356 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx" pod "3699cf25-4f57-42c8-b114-ff2fb561d189" (UID: "3699cf25-4f57-42c8-b114-ff2fb561d189"). InnerVolumeSpecName "kube-api-access-j8ccx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:31.364935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.364912 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc" pod "dfa83e29-78fd-457d-911f-1a828ec86b4e" (UID: "dfa83e29-78fd-457d-911f-1a828ec86b4e"). InnerVolumeSpecName "kube-api-access-v8lwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:31.464423 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.464390 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8lwc\" (UniqueName: \"kubernetes.io/projected/dfa83e29-78fd-457d-911f-1a828ec86b4e-kube-api-access-v8lwc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.464423 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.464425 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ccx\" (UniqueName: \"kubernetes.io/projected/3699cf25-4f57-42c8-b114-ff2fb561d189-kube-api-access-j8ccx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:31.944435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.944397 5980 generic.go:356] "Generic (PLEG): container finished" podID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerID="b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb" exitCode=0 Oct 30 06:21:31.944917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.944442 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" Oct 30 06:21:31.944917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.944476 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" event={"ID":"3699cf25-4f57-42c8-b114-ff2fb561d189","Type":"ContainerDied","Data":"b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb"} Oct 30 06:21:31.944917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.944522 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8" event={"ID":"3699cf25-4f57-42c8-b114-ff2fb561d189","Type":"ContainerDied","Data":"922735f7db065bde0844885b3c80722d43ed89826c4ac2c4a0c21406c9c99cc8"} Oct 30 06:21:31.944917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.944545 5980 scope.go:122] "RemoveContainer" containerID="b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb" Oct 30 06:21:31.946288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.946263 5980 generic.go:356] "Generic (PLEG): container finished" podID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerID="9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017" exitCode=0 Oct 30 06:21:31.946491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.946296 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" event={"ID":"dfa83e29-78fd-457d-911f-1a828ec86b4e","Type":"ContainerDied","Data":"9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017"} Oct 30 06:21:31.946619 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.946522 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" event={"ID":"dfa83e29-78fd-457d-911f-1a828ec86b4e","Type":"ContainerDied","Data":"9654531910b5dd13005b74e19d3b02befb5897e2f5fff68bac18cd063c2d492f"} Oct 30 06:21:31.946619 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.946315 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg" Oct 30 06:21:31.957331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.957310 5980 scope.go:122] "RemoveContainer" containerID="b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb" Oct 30 06:21:31.957605 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:31.957578 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb\": container with ID starting with b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb not found: ID does not exist" containerID="b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb" Oct 30 06:21:31.957761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.957617 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb"} err="failed to get container status \"b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb\": rpc error: code = NotFound desc = could not find container \"b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb\": container with ID starting with b19a29176667735aabea3fd87dd12f427b9b1877c556e1fd85cd027fa07965fb not found: ID does not exist" Oct 30 06:21:31.957761 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.957641 5980 scope.go:122] "RemoveContainer" containerID="9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017" Oct 30 06:21:31.970885 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.970845 5980 scope.go:122] "RemoveContainer" containerID="9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017" Oct 30 06:21:31.971318 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:31.971288 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017\": container with ID starting with 9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017 not found: ID does not exist" containerID="9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017" Oct 30 06:21:31.971415 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:31.971323 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017"} err="failed to get container status \"9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017\": rpc error: code = NotFound desc = could not find container \"9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017\": container with ID starting with 9f1f8bf1a346ec6880f1fc617baa342e3d6d64bdd2b8ee60a4419cfd6f188017 not found: ID does not exist" Oct 30 06:21:32.001181 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:32.001153 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg"] Oct 30 06:21:32.018895 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:32.018869 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5052f3d9-b903-410f-83ba-7b0cd797e710" path="/var/lib/kubelet/pods/5052f3d9-b903-410f-83ba-7b0cd797e710/volumes" Oct 30 06:21:32.019331 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:32.019309 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-91f42c-c8bd94999-gqdsg"] Oct 30 06:21:32.048202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:32.048167 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8"] Oct 30 06:21:32.048338 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:32.048224 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-91f42c-79f8bc77f-j8tj8"] Oct 30 06:21:34.015186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:34.015143 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" path="/var/lib/kubelet/pods/3699cf25-4f57-42c8-b114-ff2fb561d189/volumes" Oct 30 06:21:34.015749 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:34.015731 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" path="/var/lib/kubelet/pods/dfa83e29-78fd-457d-911f-1a828ec86b4e/volumes" Oct 30 06:21:43.118184 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:21:43.118147 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:21:48.992056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:48.992015 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg"] Oct 30 06:21:48.992607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:48.992296 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" containerID="cri-o://6e55585e707a0c780d6a9b945c0e72dd216ff928e8db3b957da5a5d612ded8eb" gracePeriod=30 Oct 30 06:21:54.178898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.178399 5980 generic.go:356] "Generic (PLEG): container finished" podID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerID="6e55585e707a0c780d6a9b945c0e72dd216ff928e8db3b957da5a5d612ded8eb" exitCode=0 Oct 30 06:21:54.178898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.178535 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" event={"ID":"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8","Type":"ContainerDied","Data":"6e55585e707a0c780d6a9b945c0e72dd216ff928e8db3b957da5a5d612ded8eb"} Oct 30 06:21:54.178898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.178594 5980 scope.go:122] "RemoveContainer" containerID="c528fbd23330b3b9f3168ea8220390c3daf0b57c8d26c9721c6a24738ff7adcf" Oct 30 06:21:54.299245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.299216 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:21:54.412908 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.412871 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.412920 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.412961 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413345 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413180 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413345 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413240 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413345 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413289 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:54.413512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413475 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413569 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413648 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413590 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.413709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413669 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.413824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413777 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413811 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy\") pod \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\" (UID: \"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8\") " Oct 30 06:21:54.413955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.413933 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.414372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.414342 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.414541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.414422 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.414541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.414441 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.414541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.414456 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.414541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.414470 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.416439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.416385 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:54.416557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.416515 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "kube-api-access-fwq92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:54.416557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.416548 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.417987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.417963 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token" pod "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" (UID: "e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:54.452623 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.452599 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb"] Oct 30 06:21:54.515967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.515899 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.515967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.515921 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.515967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.515931 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.515967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.515940 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwq92\" (UniqueName: \"kubernetes.io/projected/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-kube-api-access-fwq92\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.515967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:54.515949 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:21:55.187205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:55.187166 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" event={"ID":"e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8","Type":"ContainerDied","Data":"97a5fc3ebbb2383727f0b2c12c3839825536953d1728e70ba188f31bfae27343"} Oct 30 06:21:55.187647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:55.187234 5980 scope.go:122] "RemoveContainer" containerID="6e55585e707a0c780d6a9b945c0e72dd216ff928e8db3b957da5a5d612ded8eb" Oct 30 06:21:55.187647 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:55.187179 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg" Oct 30 06:21:55.247829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:55.247800 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg"] Oct 30 06:21:55.262958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:55.262930 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-stcmg"] Oct 30 06:21:56.014658 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:21:56.014628 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" path="/var/lib/kubelet/pods/e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8/volumes" Oct 30 06:22:04.271422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:04.271386 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl"] Oct 30 06:22:09.272521 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.272469 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" podUID="546784f0-9fd3-4623-b9d7-19923c40b49e" containerName="payload-pre-processing" containerID="cri-o://4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517" gracePeriod=25 Oct 30 06:22:09.480440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.477993 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" containerID="cri-o://d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213" gracePeriod=45 Oct 30 06:22:09.514059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.514027 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq"] Oct 30 06:22:09.729661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.729316 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 06:22:09.741551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.737854 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv"] Oct 30 06:22:09.741551 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.738132 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" containerID="cri-o://db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424" gracePeriod=30 Oct 30 06:22:09.766383 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.766200 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6"] Oct 30 06:22:09.767233 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.766813 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" containerID="cri-o://f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5" gracePeriod=30 Oct 30 06:22:09.791427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.778568 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume\") pod \"546784f0-9fd3-4623-b9d7-19923c40b49e\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " Oct 30 06:22:09.791427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.778694 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4\" (UniqueName: \"kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4\") pod \"546784f0-9fd3-4623-b9d7-19923c40b49e\" (UID: \"546784f0-9fd3-4623-b9d7-19923c40b49e\") " Oct 30 06:22:09.791427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.786955 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume" pod "546784f0-9fd3-4623-b9d7-19923c40b49e" (UID: "546784f0-9fd3-4623-b9d7-19923c40b49e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:09.791427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.790616 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4" pod "546784f0-9fd3-4623-b9d7-19923c40b49e" (UID: "546784f0-9fd3-4623-b9d7-19923c40b49e"). InnerVolumeSpecName "kube-api-access-4qvz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:09.799473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.798846 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/546784f0-9fd3-4623-b9d7-19923c40b49e-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:09.799473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.798873 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qvz4\" (UniqueName: \"kubernetes.io/projected/546784f0-9fd3-4623-b9d7-19923c40b49e-kube-api-access-4qvz4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:09.819163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.805297 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw"] Oct 30 06:22:09.819163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.805540 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" containerID="cri-o://c62c9fd49d351281d0c9605d104dd2fcb35967bc043f4fad89fd60eacaccc81d" gracePeriod=30 Oct 30 06:22:09.819163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.808696 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn"] Oct 30 06:22:09.819163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:09.808930 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" containerID="cri-o://55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc" gracePeriod=30 Oct 30 06:22:10.109932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.108970 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 06:22:10.238253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238216 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs\" (UniqueName: \"kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs\") pod \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " Oct 30 06:22:10.238439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238363 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache\") pod \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " Oct 30 06:22:10.238439 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238429 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm\") pod \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " Oct 30 06:22:10.238604 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238581 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home\") pod \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " Oct 30 06:22:10.238658 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238621 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5\") pod \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " Oct 30 06:22:10.238720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238682 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache" pod "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" (UID: "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.238720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238693 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir\") pod \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\" (UID: \"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342\") " Oct 30 06:22:10.238983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.238900 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir" pod "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" (UID: "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.239223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.239194 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home" pod "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" (UID: "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.239926 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.239901 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.240037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.239932 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.240037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.239946 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.240821 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.240803 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm" pod "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" (UID: "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.240895 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.240831 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs" pod "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" (UID: "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:10.241060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.241044 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5" pod "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" (UID: "1605026c-45f3-4dfc-a9a5-6a9ebf5ce342"). InnerVolumeSpecName "kube-api-access-49gg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:10.319983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.318385 5980 generic.go:356] "Generic (PLEG): container finished" podID="546784f0-9fd3-4623-b9d7-19923c40b49e" containerID="4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517" exitCode=0 Oct 30 06:22:10.319983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.318542 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" event={"ID":"546784f0-9fd3-4623-b9d7-19923c40b49e","Type":"ContainerDied","Data":"4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517"} Oct 30 06:22:10.319983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.318599 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" event={"ID":"546784f0-9fd3-4623-b9d7-19923c40b49e","Type":"ContainerDied","Data":"ff8b856e23824205a0f36db57d16fef10b750a6a70d5e8a599c8d691d3787ea8"} Oct 30 06:22:10.319983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.318629 5980 scope.go:122] "RemoveContainer" containerID="4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517" Oct 30 06:22:10.319983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.318805 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl" Oct 30 06:22:10.324685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.323417 5980 generic.go:356] "Generic (PLEG): container finished" podID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerID="d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213" exitCode=0 Oct 30 06:22:10.324685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.323495 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" event={"ID":"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342","Type":"ContainerDied","Data":"d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213"} Oct 30 06:22:10.324685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.323523 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" event={"ID":"1605026c-45f3-4dfc-a9a5-6a9ebf5ce342","Type":"ContainerDied","Data":"dd988a6fa6cb5d0b61a201f2168c7642df397a702e3d6f0e66bf888b46db2a9a"} Oct 30 06:22:10.324685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.323587 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb" Oct 30 06:22:10.342193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.342158 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.342193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.342193 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.342342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.342207 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gg5\" (UniqueName: \"kubernetes.io/projected/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342-kube-api-access-49gg5\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.345909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.345883 5980 scope.go:122] "RemoveContainer" containerID="4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517" Oct 30 06:22:10.346322 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:10.346296 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517\": container with ID starting with 4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517 not found: ID does not exist" containerID="4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517" Oct 30 06:22:10.346416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.346341 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517"} err="failed to get container status \"4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517\": rpc error: code = NotFound desc = could not find container \"4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517\": container with ID starting with 4b66fc325d61ad0d80f69267a960786b2ffb542c799aec8735e974682ad3d517 not found: ID does not exist" Oct 30 06:22:10.346416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.346365 5980 scope.go:122] "RemoveContainer" containerID="d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213" Oct 30 06:22:10.360730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.360686 5980 scope.go:122] "RemoveContainer" containerID="d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213" Oct 30 06:22:10.361189 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:10.361049 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213\": container with ID starting with d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213 not found: ID does not exist" containerID="d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213" Oct 30 06:22:10.361189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.361086 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213"} err="failed to get container status \"d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213\": rpc error: code = NotFound desc = could not find container \"d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213\": container with ID starting with d1b4ada11ea2dacc1e252f7c2fa8dc5c84610adb7af82b9ee3c8923fd6d5c213 not found: ID does not exist" Oct 30 06:22:10.395888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.395853 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl"] Oct 30 06:22:10.403070 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.403037 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-f22a52-7d54f66bcf-nsqhl"] Oct 30 06:22:10.436540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.436502 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb"] Oct 30 06:22:10.440740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.440714 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-2-w5-a35835-kserve-846d7c6b86-bfcdb"] Oct 30 06:22:10.957607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.957568 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:10.957795 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:10.957650 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:12.025358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:12.025320 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" path="/var/lib/kubelet/pods/1605026c-45f3-4dfc-a9a5-6a9ebf5ce342/volumes" Oct 30 06:22:12.025785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:12.025770 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546784f0-9fd3-4623-b9d7-19923c40b49e" path="/var/lib/kubelet/pods/546784f0-9fd3-4623-b9d7-19923c40b49e/volumes" Oct 30 06:22:12.819132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:12.818982 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd"] Oct 30 06:22:12.819333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:12.819218 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" podUID="3d31a88e-bac3-4483-85bd-146ab72eb228" containerName="maas-api" containerID="cri-o://2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d" gracePeriod=30 Oct 30 06:22:13.219009 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:13.218981 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:22:13.237542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.237517 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 06:22:13.364134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.349697 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4\" (UniqueName: \"kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4\") pod \"3d31a88e-bac3-4483-85bd-146ab72eb228\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " Oct 30 06:22:13.364134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.349877 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls\") pod \"3d31a88e-bac3-4483-85bd-146ab72eb228\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " Oct 30 06:22:13.364134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.350007 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs\") pod \"3d31a88e-bac3-4483-85bd-146ab72eb228\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " Oct 30 06:22:13.364134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.350053 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca\") pod \"3d31a88e-bac3-4483-85bd-146ab72eb228\" (UID: \"3d31a88e-bac3-4483-85bd-146ab72eb228\") " Oct 30 06:22:13.370199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.364836 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca" pod "3d31a88e-bac3-4483-85bd-146ab72eb228" (UID: "3d31a88e-bac3-4483-85bd-146ab72eb228"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:13.430489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.404297 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls" pod "3d31a88e-bac3-4483-85bd-146ab72eb228" (UID: "3d31a88e-bac3-4483-85bd-146ab72eb228"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:13.430489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.404328 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs" pod "3d31a88e-bac3-4483-85bd-146ab72eb228" (UID: "3d31a88e-bac3-4483-85bd-146ab72eb228"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:13.430489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.404424 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4" pod "3d31a88e-bac3-4483-85bd-146ab72eb228" (UID: "3d31a88e-bac3-4483-85bd-146ab72eb228"). InnerVolumeSpecName "kube-api-access-m8kt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:13.448563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.432688 5980 generic.go:356] "Generic (PLEG): container finished" podID="3d31a88e-bac3-4483-85bd-146ab72eb228" containerID="2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d" exitCode=0 Oct 30 06:22:13.448563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.432739 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" event={"ID":"3d31a88e-bac3-4483-85bd-146ab72eb228","Type":"ContainerDied","Data":"2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d"} Oct 30 06:22:13.448563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.432803 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" event={"ID":"3d31a88e-bac3-4483-85bd-146ab72eb228","Type":"ContainerDied","Data":"013eb6b61253c36ff0878fd8863839f6e1de491710cd6702517146d3eebb81cd"} Oct 30 06:22:13.448563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.432836 5980 scope.go:122] "RemoveContainer" containerID="2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d" Oct 30 06:22:13.448563 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.432981 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd" Oct 30 06:22:13.452634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.451903 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:13.452634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.451928 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d31a88e-bac3-4483-85bd-146ab72eb228-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:13.452634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.451938 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d31a88e-bac3-4483-85bd-146ab72eb228-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:13.452634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.451948 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8kt4\" (UniqueName: \"kubernetes.io/projected/3d31a88e-bac3-4483-85bd-146ab72eb228-kube-api-access-m8kt4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:13.462865 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.462486 5980 scope.go:122] "RemoveContainer" containerID="2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d" Oct 30 06:22:13.462865 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:13.462818 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d\": container with ID starting with 2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d not found: ID does not exist" containerID="2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d" Oct 30 06:22:13.463085 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.462858 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d"} err="failed to get container status \"2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d\": rpc error: code = NotFound desc = could not find container \"2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d\": container with ID starting with 2813b870cdb872ee9f56d634e4248707957f00314758fc92f4802c4ba26cde2d not found: ID does not exist" Oct 30 06:22:13.505676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.505647 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd"] Oct 30 06:22:13.514186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:13.514161 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-197e9b-6c9f984757-fpzqd"] Oct 30 06:22:14.017270 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:14.017236 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d31a88e-bac3-4483-85bd-146ab72eb228" path="/var/lib/kubelet/pods/3d31a88e-bac3-4483-85bd-146ab72eb228/volumes" Oct 30 06:22:14.888064 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:14.888032 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc"] Oct 30 06:22:14.888504 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:14.888305 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" containerID="cri-o://e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b" gracePeriod=30 Oct 30 06:22:15.237133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.230750 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:22:15.309178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.304962 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.309178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.308022 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.309178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.308232 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.310243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.310191 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.314941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.314909 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.333046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.318157 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.333046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.329209 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.333046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.329282 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.333046 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.329370 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.333673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.333394 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.334676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.334650 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.334855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.334833 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.334957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.334937 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld\") pod \"950d117c-c5ac-4284-b761-270f957da6c1\" (UID: \"950d117c-c5ac-4284-b761-270f957da6c1\") " Oct 30 06:22:15.335620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.335587 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.336873 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.336845 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/950d117c-c5ac-4284-b761-270f957da6c1-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.336975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.336876 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.336975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.336888 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.336975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.336897 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.336975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.336906 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.342173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.339270 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "kube-api-access-vl2ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.344944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.344906 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.348162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.348129 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.350485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.350187 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy" pod "950d117c-c5ac-4284-b761-270f957da6c1" (UID: "950d117c-c5ac-4284-b761-270f957da6c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.371649 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.371375 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:22:15.374286 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.373684 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:22:15.438633 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438593 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.438813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438660 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.438813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438721 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.438813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438755 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.438813 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438795 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438842 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438901 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.439026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438930 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438948 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.438986 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439012 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439032 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.439325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439061 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439130 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs\") pod \"45725349-ec16-493f-a1e0-85e3db07452d\" (UID: \"45725349-ec16-493f-a1e0-85e3db07452d\") " Oct 30 06:22:15.439325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439157 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.439325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439280 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.439325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439319 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.439745 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.439349 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo\") pod \"921ed1c2-f38e-4842-8fa7-104af4a9330a\" (UID: \"921ed1c2-f38e-4842-8fa7-104af4a9330a\") " Oct 30 06:22:15.440133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.440082 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/950d117c-c5ac-4284-b761-270f957da6c1-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.447011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.446984 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/950d117c-c5ac-4284-b761-270f957da6c1-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.447014 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.447031 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2ld\" (UniqueName: \"kubernetes.io/projected/950d117c-c5ac-4284-b761-270f957da6c1-kube-api-access-vl2ld\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.440495 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.441992 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.442397 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.442728 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.443050 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.443575 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.444144 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.446260 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.446523 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.447160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.446705 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.453320 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.453289 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.453446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.453396 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.453525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.453503 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "kube-api-access-sc9xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.456187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.456099 5980 generic.go:356] "Generic (PLEG): container finished" podID="45725349-ec16-493f-a1e0-85e3db07452d" containerID="55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc" exitCode=0 Oct 30 06:22:15.456364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.456273 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" event={"ID":"45725349-ec16-493f-a1e0-85e3db07452d","Type":"ContainerDied","Data":"55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc"} Oct 30 06:22:15.456364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.456330 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" event={"ID":"45725349-ec16-493f-a1e0-85e3db07452d","Type":"ContainerDied","Data":"ef3453270887395e5b3e46259672448fe2c8b633c7eef52149424375c8f46f65"} Oct 30 06:22:15.456364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.456337 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn" Oct 30 06:22:15.456364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.456361 5980 scope.go:122] "RemoveContainer" containerID="55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc" Oct 30 06:22:15.460318 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.460288 5980 generic.go:356] "Generic (PLEG): container finished" podID="950d117c-c5ac-4284-b761-270f957da6c1" containerID="db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424" exitCode=0 Oct 30 06:22:15.460469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.460446 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" event={"ID":"950d117c-c5ac-4284-b761-270f957da6c1","Type":"ContainerDied","Data":"db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424"} Oct 30 06:22:15.460582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.460493 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" event={"ID":"950d117c-c5ac-4284-b761-270f957da6c1","Type":"ContainerDied","Data":"1baaf49c15d5e947426c9df7d4f499c1e68c16a32676aac09eaf93f416c132c5"} Oct 30 06:22:15.460582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.460576 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv" Oct 30 06:22:15.465210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.464902 5980 generic.go:356] "Generic (PLEG): container finished" podID="377612ce-6b65-4347-949f-d03e31a4c596" containerID="c62c9fd49d351281d0c9605d104dd2fcb35967bc043f4fad89fd60eacaccc81d" exitCode=0 Oct 30 06:22:15.465210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.465075 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" event={"ID":"377612ce-6b65-4347-949f-d03e31a4c596","Type":"ContainerDied","Data":"c62c9fd49d351281d0c9605d104dd2fcb35967bc043f4fad89fd60eacaccc81d"} Oct 30 06:22:15.468268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.468237 5980 generic.go:356] "Generic (PLEG): container finished" podID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerID="f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5" exitCode=0 Oct 30 06:22:15.468493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.468308 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" event={"ID":"921ed1c2-f38e-4842-8fa7-104af4a9330a","Type":"ContainerDied","Data":"f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5"} Oct 30 06:22:15.468493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.468351 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" event={"ID":"921ed1c2-f38e-4842-8fa7-104af4a9330a","Type":"ContainerDied","Data":"f22ea79638905297df157ff1260769c48d678cc14339bb7bc6669bb2b59863c2"} Oct 30 06:22:15.468493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.468428 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6" Oct 30 06:22:15.481291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.481259 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.481820 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.481776 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.482182 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.482153 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.482886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.482864 5980 scope.go:122] "RemoveContainer" containerID="55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc" Oct 30 06:22:15.483378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.482972 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token" pod "921ed1c2-f38e-4842-8fa7-104af4a9330a" (UID: "921ed1c2-f38e-4842-8fa7-104af4a9330a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.484219 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.484179 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s" pod "45725349-ec16-493f-a1e0-85e3db07452d" (UID: "45725349-ec16-493f-a1e0-85e3db07452d"). InnerVolumeSpecName "kube-api-access-k8q8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.485346 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:15.485176 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc\": container with ID starting with 55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc not found: ID does not exist" containerID="55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc" Oct 30 06:22:15.485346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.485215 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc"} err="failed to get container status \"55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc\": rpc error: code = NotFound desc = could not find container \"55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc\": container with ID starting with 55a2c882fe168c6d916fe5c9003a3056bd32b6b8dd89b62952b08d64376dd1bc not found: ID does not exist" Oct 30 06:22:15.485346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.485242 5980 scope.go:122] "RemoveContainer" containerID="db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424" Oct 30 06:22:15.491489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.491465 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:22:15.500396 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.500374 5980 scope.go:122] "RemoveContainer" containerID="db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424" Oct 30 06:22:15.500717 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:15.500697 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424\": container with ID starting with db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424 not found: ID does not exist" containerID="db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424" Oct 30 06:22:15.500874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.500728 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424"} err="failed to get container status \"db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424\": rpc error: code = NotFound desc = could not find container \"db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424\": container with ID starting with db3e8ffb6a46fe76a26122c333cf845595b158ec9571d5910e0db4cd0272e424 not found: ID does not exist" Oct 30 06:22:15.500874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.500764 5980 scope.go:122] "RemoveContainer" containerID="30107178ffb80b3edd9b8257918d65a04c5835ca07e68ca109bcd14ed6bed061" Oct 30 06:22:15.520264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.520189 5980 scope.go:122] "RemoveContainer" containerID="f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5" Oct 30 06:22:15.537056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.537033 5980 scope.go:122] "RemoveContainer" containerID="f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5" Oct 30 06:22:15.538279 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:15.538250 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5\": container with ID starting with f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5 not found: ID does not exist" containerID="f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5" Oct 30 06:22:15.538448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.538416 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5"} err="failed to get container status \"f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5\": rpc error: code = NotFound desc = could not find container \"f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5\": container with ID starting with f9b20b3600c09fa4b153cbc72330409f5000eb80d44cf15e6d73fd768f9b53e5 not found: ID does not exist" Oct 30 06:22:15.548582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.548551 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.548840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.548818 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.550537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.548846 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.550609 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.550699 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.550746 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.550864 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.550916 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551010 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.551422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551080 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert\") pod \"377612ce-6b65-4347-949f-d03e31a4c596\" (UID: \"377612ce-6b65-4347-949f-d03e31a4c596\") " Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551871 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551898 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551915 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8q8s\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-kube-api-access-k8q8s\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551930 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45725349-ec16-493f-a1e0-85e3db07452d-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551946 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551960 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45725349-ec16-493f-a1e0-85e3db07452d-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551974 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.551987 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45725349-ec16-493f-a1e0-85e3db07452d-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552001 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552015 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9xc\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-kube-api-access-sc9xc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552031 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552047 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552064 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45725349-ec16-493f-a1e0-85e3db07452d-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.552253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552078 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.553464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552225 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.553464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552614 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.553464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552632 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/921ed1c2-f38e-4842-8fa7-104af4a9330a-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.553464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552645 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/921ed1c2-f38e-4842-8fa7-104af4a9330a-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.553464 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552659 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/921ed1c2-f38e-4842-8fa7-104af4a9330a-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.553900 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.550913 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.553900 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.552994 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.553900 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.553305 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.554545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.554518 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.556931 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.556892 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "kube-api-access-fp77c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.557538 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.557508 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.558767 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.558745 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.559912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.559852 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token" pod "377612ce-6b65-4347-949f-d03e31a4c596" (UID: "377612ce-6b65-4347-949f-d03e31a4c596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.590765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.590728 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv"] Oct 30 06:22:15.594279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.594248 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-8jscv"] Oct 30 06:22:15.654575 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654541 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654577 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654594 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654609 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654624 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377612ce-6b65-4347-949f-d03e31a4c596-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654638 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp77c\" (UniqueName: \"kubernetes.io/projected/377612ce-6b65-4347-949f-d03e31a4c596-kube-api-access-fp77c\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654651 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377612ce-6b65-4347-949f-d03e31a4c596-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.654726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.654665 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377612ce-6b65-4347-949f-d03e31a4c596-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.828421 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.828392 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn"] Oct 30 06:22:15.892582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.892540 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tt6rn"] Oct 30 06:22:15.950369 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.950337 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6"] Oct 30 06:22:15.959160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:15.959101 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-tp2k6"] Oct 30 06:22:16.015681 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.015643 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45725349-ec16-493f-a1e0-85e3db07452d" path="/var/lib/kubelet/pods/45725349-ec16-493f-a1e0-85e3db07452d/volumes" Oct 30 06:22:16.016295 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.016279 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" path="/var/lib/kubelet/pods/921ed1c2-f38e-4842-8fa7-104af4a9330a/volumes" Oct 30 06:22:16.016866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.016850 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950d117c-c5ac-4284-b761-270f957da6c1" path="/var/lib/kubelet/pods/950d117c-c5ac-4284-b761-270f957da6c1/volumes" Oct 30 06:22:16.476454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.476427 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" Oct 30 06:22:16.476750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.476431 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw" event={"ID":"377612ce-6b65-4347-949f-d03e31a4c596","Type":"ContainerDied","Data":"7b21974cee7b0d9d4671380844e6341125ec72aa50de0ebfb5c787a79f551e59"} Oct 30 06:22:16.476891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.476796 5980 scope.go:122] "RemoveContainer" containerID="c62c9fd49d351281d0c9605d104dd2fcb35967bc043f4fad89fd60eacaccc81d" Oct 30 06:22:16.568741 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.568706 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw"] Oct 30 06:22:16.584845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.584806 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-n8wdw"] Oct 30 06:22:16.840787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.840749 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d"] Oct 30 06:22:16.841415 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841400 5980 cpu_manager.go:414] "containerMap: removing container" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841417 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841430 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841438 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841451 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841457 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7294cfd8-d116-49bf-9361-a2778bea6118" containerName="istio-proxy" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841466 5980 cpu_manager.go:414] "containerMap: removing container" podUID="cc0e8018-070f-4add-ab40-569613d1effa" containerName="payload-processing" Oct 30 06:22:16.841477 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841471 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0e8018-070f-4add-ab40-569613d1effa" containerName="payload-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841482 5980 cpu_manager.go:414] "containerMap: removing container" podUID="4896357c-db54-4af4-9c1c-d08559326b79" containerName="maas-api" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841487 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4896357c-db54-4af4-9c1c-d08559326b79" containerName="maas-api" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841493 5980 cpu_manager.go:414] "containerMap: removing container" podUID="07a48d1d-d63f-4a65-856d-2487f232727f" containerName="payload-pre-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841497 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a48d1d-d63f-4a65-856d-2487f232727f" containerName="payload-pre-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841506 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841510 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841533 5980 cpu_manager.go:414] "containerMap: removing container" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerName="payload-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841538 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerName="payload-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841544 5980 cpu_manager.go:414] "containerMap: removing container" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" containerName="test-curl-9204-d5ade7" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841550 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" containerName="test-curl-9204-d5ade7" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841557 5980 cpu_manager.go:414] "containerMap: removing container" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841561 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841569 5980 cpu_manager.go:414] "containerMap: removing container" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841573 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841583 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerName="payload-pre-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841587 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerName="payload-pre-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841595 5980 cpu_manager.go:414] "containerMap: removing container" podUID="3d31a88e-bac3-4483-85bd-146ab72eb228" containerName="maas-api" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841601 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d31a88e-bac3-4483-85bd-146ab72eb228" containerName="maas-api" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841614 5980 cpu_manager.go:414] "containerMap: removing container" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841619 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841626 5980 cpu_manager.go:414] "containerMap: removing container" podUID="546784f0-9fd3-4623-b9d7-19923c40b49e" containerName="payload-pre-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841631 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546784f0-9fd3-4623-b9d7-19923c40b49e" containerName="payload-pre-processing" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841647 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5052f3d9-b903-410f-83ba-7b0cd797e710" containerName="main" Oct 30 06:22:16.841693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841651 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5052f3d9-b903-410f-83ba-7b0cd797e710" containerName="main" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841739 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa83e29-78fd-457d-911f-1a828ec86b4e" containerName="payload-processing" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841749 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="88af9da4-0208-4014-84ba-d86f3ed1e7d0" containerName="test-curl-9204-d5ade7" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841761 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="45725349-ec16-493f-a1e0-85e3db07452d" containerName="istio-proxy" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841770 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841777 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3699cf25-4f57-42c8-b114-ff2fb561d189" containerName="payload-pre-processing" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841783 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d31a88e-bac3-4483-85bd-146ab72eb228" containerName="maas-api" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841790 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="950d117c-c5ac-4284-b761-270f957da6c1" containerName="istio-proxy" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841796 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="546784f0-9fd3-4623-b9d7-19923c40b49e" containerName="payload-pre-processing" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841802 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841808 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5052f3d9-b903-410f-83ba-7b0cd797e710" containerName="main" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841813 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841819 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="921ed1c2-f38e-4842-8fa7-104af4a9330a" containerName="istio-proxy" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841826 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1605026c-45f3-4dfc-a9a5-6a9ebf5ce342" containerName="main" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841831 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0e8018-070f-4add-ab40-569613d1effa" containerName="payload-processing" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841836 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4896357c-db54-4af4-9c1c-d08559326b79" containerName="maas-api" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841841 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a48d1d-d63f-4a65-856d-2487f232727f" containerName="payload-pre-processing" Oct 30 06:22:16.842444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.841845 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" Oct 30 06:22:16.846902 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.846886 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:16.864383 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.864356 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1685dd\"" Oct 30 06:22:16.864637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.864621 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1685dd-dockercfg-dhn7z\"" Oct 30 06:22:16.867920 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.867900 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d"] Oct 30 06:22:16.993170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.984253 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:16.993858 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:16.993829 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66qxw\" (UniqueName: \"kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:17.113010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.112912 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66qxw\" (UniqueName: \"kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:17.114693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.114666 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:17.115934 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.115815 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:17.138704 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.138671 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66qxw\" (UniqueName: \"kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:17.161753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.161733 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:17.479680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.479648 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d"] Oct 30 06:22:17.479988 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:22:17.479960 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14d1d94_d361_4634_9cbc_cf4621f87cc1.slice/crio-f1f8a5a5fc61094bfe27d33a0f203254c6677d86f93b662417cac36899521974 WatchSource:0}: Error finding container f1f8a5a5fc61094bfe27d33a0f203254c6677d86f93b662417cac36899521974: Status 404 returned error can't find the container with id f1f8a5a5fc61094bfe27d33a0f203254c6677d86f93b662417cac36899521974 Oct 30 06:22:17.490168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:17.490138 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" event={"ID":"a14d1d94-d361-4634-9cbc-cf4621f87cc1","Type":"ContainerStarted","Data":"f1f8a5a5fc61094bfe27d33a0f203254c6677d86f93b662417cac36899521974"} Oct 30 06:22:18.014081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:18.014048 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377612ce-6b65-4347-949f-d03e31a4c596" path="/var/lib/kubelet/pods/377612ce-6b65-4347-949f-d03e31a4c596/volumes" Oct 30 06:22:18.498881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:18.498843 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" event={"ID":"a14d1d94-d361-4634-9cbc-cf4621f87cc1","Type":"ContainerStarted","Data":"c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d"} Oct 30 06:22:18.499071 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:18.498981 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:18.544541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:18.544498 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" podStartSLOduration=2.544484605 podStartE2EDuration="2.544484605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:18.543907944 +0000 UTC m=+2992.651946978" watchObservedRunningTime="2026-10-30 06:22:18.544484605 +0000 UTC m=+2992.652523637" Oct 30 06:22:20.357873 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.357843 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:22:20.458402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458368 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458490 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458561 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458617 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458676 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458740 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458846 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458822 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.458960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458923 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.459249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.459011 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.459249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.458929 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.459418 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.459317 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket\") pod \"26a18b6a-88bf-43fd-b7a8-abae1840013d\" (UID: \"26a18b6a-88bf-43fd-b7a8-abae1840013d\") " Oct 30 06:22:20.460048 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.459639 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:20.460048 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.459867 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.460618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.460479 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.460618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.460537 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a18b6a-88bf-43fd-b7a8-abae1840013d-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.460618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.460566 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.460618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.460581 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.460618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.460596 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.461516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.461488 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "kube-api-access-5h5x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:20.461652 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.461630 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.462242 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.462216 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:20.463193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.463169 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token" pod "26a18b6a-88bf-43fd-b7a8-abae1840013d" (UID: "26a18b6a-88bf-43fd-b7a8-abae1840013d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:20.516204 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.516177 5980 generic.go:356] "Generic (PLEG): container finished" podID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerID="e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b" exitCode=0 Oct 30 06:22:20.516325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.516227 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" Oct 30 06:22:20.516325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.516250 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" event={"ID":"26a18b6a-88bf-43fd-b7a8-abae1840013d","Type":"ContainerDied","Data":"e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b"} Oct 30 06:22:20.516325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.516290 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc" event={"ID":"26a18b6a-88bf-43fd-b7a8-abae1840013d","Type":"ContainerDied","Data":"578f29ddd8effaa0ecefaf3132ca5588194bcf597f5ea038a621d2c6412efbfd"} Oct 30 06:22:20.516325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.516309 5980 scope.go:122] "RemoveContainer" containerID="e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b" Oct 30 06:22:20.527468 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.527446 5980 scope.go:122] "RemoveContainer" containerID="e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b" Oct 30 06:22:20.527752 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:20.527730 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b\": container with ID starting with e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b not found: ID does not exist" containerID="e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b" Oct 30 06:22:20.528302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.527762 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b"} err="failed to get container status \"e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b\": rpc error: code = NotFound desc = could not find container \"e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b\": container with ID starting with e6e8fa03384cd1fc767ae36fd886815adb5cb8f0a01d1fec1cc213bf0b85a37b not found: ID does not exist" Oct 30 06:22:20.561848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.561826 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.561848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.561849 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.561979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.561858 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.561979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.561865 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5x9\" (UniqueName: \"kubernetes.io/projected/26a18b6a-88bf-43fd-b7a8-abae1840013d-kube-api-access-5h5x9\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.561979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.561875 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a18b6a-88bf-43fd-b7a8-abae1840013d-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.576170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.576146 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc"] Oct 30 06:22:20.585576 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:20.585510 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-ns8rc"] Oct 30 06:22:22.018884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:22.018849 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" path="/var/lib/kubelet/pods/26a18b6a-88bf-43fd-b7a8-abae1840013d/volumes" Oct 30 06:22:24.557232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.557195 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" podUID="39ed193f-7ac3-4428-9051-59a8e1b62b4e" containerName="main" containerID="cri-o://bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a" gracePeriod=45 Oct 30 06:22:24.854172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.854148 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:22:24.954601 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954568 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir\") pod \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " Oct 30 06:22:24.954801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954625 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home\") pod \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " Oct 30 06:22:24.954801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954675 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm\") pod \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " Oct 30 06:22:24.954801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954766 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t\" (UniqueName: \"kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t\") pod \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " Oct 30 06:22:24.954991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954839 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir" pod "39ed193f-7ac3-4428-9051-59a8e1b62b4e" (UID: "39ed193f-7ac3-4428-9051-59a8e1b62b4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:24.954991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954849 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs\") pod \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " Oct 30 06:22:24.954991 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.954944 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home" pod "39ed193f-7ac3-4428-9051-59a8e1b62b4e" (UID: "39ed193f-7ac3-4428-9051-59a8e1b62b4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:24.955191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.955028 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache\") pod \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\" (UID: \"39ed193f-7ac3-4428-9051-59a8e1b62b4e\") " Oct 30 06:22:24.955329 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.955305 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache" pod "39ed193f-7ac3-4428-9051-59a8e1b62b4e" (UID: "39ed193f-7ac3-4428-9051-59a8e1b62b4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:24.955950 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.955925 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:24.956054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.955980 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:24.956054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.955995 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:24.957400 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.957378 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs" pod "39ed193f-7ac3-4428-9051-59a8e1b62b4e" (UID: "39ed193f-7ac3-4428-9051-59a8e1b62b4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:24.957515 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.957479 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm" pod "39ed193f-7ac3-4428-9051-59a8e1b62b4e" (UID: "39ed193f-7ac3-4428-9051-59a8e1b62b4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:24.957578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:24.957520 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t" pod "39ed193f-7ac3-4428-9051-59a8e1b62b4e" (UID: "39ed193f-7ac3-4428-9051-59a8e1b62b4e"). InnerVolumeSpecName "kube-api-access-sj74t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:25.057722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.057692 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ed193f-7ac3-4428-9051-59a8e1b62b4e-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.057722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.057721 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj74t\" (UniqueName: \"kubernetes.io/projected/39ed193f-7ac3-4428-9051-59a8e1b62b4e-kube-api-access-sj74t\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.057904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.057737 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ed193f-7ac3-4428-9051-59a8e1b62b4e-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.553843 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.553809 5980 generic.go:356] "Generic (PLEG): container finished" podID="39ed193f-7ac3-4428-9051-59a8e1b62b4e" containerID="bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a" exitCode=0 Oct 30 06:22:25.554031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.553907 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" event={"ID":"39ed193f-7ac3-4428-9051-59a8e1b62b4e","Type":"ContainerDied","Data":"bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a"} Oct 30 06:22:25.554031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.553919 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" Oct 30 06:22:25.554031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.553950 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq" event={"ID":"39ed193f-7ac3-4428-9051-59a8e1b62b4e","Type":"ContainerDied","Data":"4c1ca1192bf75e4cd270f610835f518d60b96dcbc02e301a16dc62684192bce7"} Oct 30 06:22:25.554031 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.553973 5980 scope.go:122] "RemoveContainer" containerID="bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a" Oct 30 06:22:25.578309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.578285 5980 scope.go:122] "RemoveContainer" containerID="bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a" Oct 30 06:22:25.578741 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:25.578715 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a\": container with ID starting with bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a not found: ID does not exist" containerID="bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a" Oct 30 06:22:25.578810 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.578754 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a"} err="failed to get container status \"bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a\": rpc error: code = NotFound desc = could not find container \"bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a\": container with ID starting with bc0368e948104e885ba3ea40249e592e33d5d9da47430b55c04196e867c0702a not found: ID does not exist" Oct 30 06:22:25.625614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.625587 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq"] Oct 30 06:22:25.639816 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:25.639790 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-f6302c/auth-test-model-f6302c-kserve-844cb99849-kghfq"] Oct 30 06:22:26.015679 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:26.015647 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ed193f-7ac3-4428-9051-59a8e1b62b4e" path="/var/lib/kubelet/pods/39ed193f-7ac3-4428-9051-59a8e1b62b4e/volumes" Oct 30 06:22:29.548134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:29.545300 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:22:38.536734 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.527865 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8"] Oct 30 06:22:38.536734 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.534364 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" podUID="b41fd892-f798-4f67-8427-65f1601efa58" containerName="maas-api" containerID="cri-o://6e28428d85565845f28ab7bcf32a1132b1109f974032efd1d280b393884f495e" gracePeriod=30 Oct 30 06:22:38.738099 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.738054 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" podUID="b41fd892-f798-4f67-8427-65f1601efa58" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.76:8443/health\": dial tcp 10.134.0.76:8443: connect: connection refused" Oct 30 06:22:38.784142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.784075 5980 generic.go:356] "Generic (PLEG): container finished" podID="b41fd892-f798-4f67-8427-65f1601efa58" containerID="6e28428d85565845f28ab7bcf32a1132b1109f974032efd1d280b393884f495e" exitCode=0 Oct 30 06:22:38.784306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.784262 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" event={"ID":"b41fd892-f798-4f67-8427-65f1601efa58","Type":"ContainerDied","Data":"6e28428d85565845f28ab7bcf32a1132b1109f974032efd1d280b393884f495e"} Oct 30 06:22:38.895887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.895857 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 06:22:38.925422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.925399 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz\" (UniqueName: \"kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz\") pod \"b41fd892-f798-4f67-8427-65f1601efa58\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " Oct 30 06:22:38.925579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.925465 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs\") pod \"b41fd892-f798-4f67-8427-65f1601efa58\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " Oct 30 06:22:38.925579 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.925512 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca\") pod \"b41fd892-f798-4f67-8427-65f1601efa58\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " Oct 30 06:22:38.925667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.925599 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls\") pod \"b41fd892-f798-4f67-8427-65f1601efa58\" (UID: \"b41fd892-f798-4f67-8427-65f1601efa58\") " Oct 30 06:22:38.925960 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.925934 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca" pod "b41fd892-f798-4f67-8427-65f1601efa58" (UID: "b41fd892-f798-4f67-8427-65f1601efa58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:38.926208 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.926146 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41fd892-f798-4f67-8427-65f1601efa58-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:38.927850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.927814 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs" pod "b41fd892-f798-4f67-8427-65f1601efa58" (UID: "b41fd892-f798-4f67-8427-65f1601efa58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:38.927850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.927829 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls" pod "b41fd892-f798-4f67-8427-65f1601efa58" (UID: "b41fd892-f798-4f67-8427-65f1601efa58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:38.928036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:38.928020 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz" pod "b41fd892-f798-4f67-8427-65f1601efa58" (UID: "b41fd892-f798-4f67-8427-65f1601efa58"). InnerVolumeSpecName "kube-api-access-kx4lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:39.027995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.027964 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:39.027995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.027988 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4lz\" (UniqueName: \"kubernetes.io/projected/b41fd892-f798-4f67-8427-65f1601efa58-kube-api-access-kx4lz\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:39.027995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.027998 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41fd892-f798-4f67-8427-65f1601efa58-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:39.796257 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.796215 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" event={"ID":"b41fd892-f798-4f67-8427-65f1601efa58","Type":"ContainerDied","Data":"cf0883c559d112a10d62283bfabdacf71d697b36162c783db097dc2e8d1838e7"} Oct 30 06:22:39.796675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.796287 5980 scope.go:122] "RemoveContainer" containerID="6e28428d85565845f28ab7bcf32a1132b1109f974032efd1d280b393884f495e" Oct 30 06:22:39.796675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.796439 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8" Oct 30 06:22:39.857903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.857872 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8"] Oct 30 06:22:39.871765 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:39.871735 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6b5b1e-5546447df9-sr7n8"] Oct 30 06:22:40.014267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:40.014235 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41fd892-f798-4f67-8427-65f1601efa58" path="/var/lib/kubelet/pods/b41fd892-f798-4f67-8427-65f1601efa58/volumes" Oct 30 06:22:43.088546 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:43.088512 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:22:47.380201 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:47.380167 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d"] Oct 30 06:22:47.380715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:47.380402 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" containerID="cri-o://c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c" gracePeriod=30 Oct 30 06:22:47.393304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:47.393275 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j"] Oct 30 06:22:47.393552 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:47.393528 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" containerID="cri-o://e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1" gracePeriod=30 Oct 30 06:22:48.193847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:48.193813 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8"] Oct 30 06:22:48.194163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:48.194100 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" containerID="cri-o://16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0" gracePeriod=30 Oct 30 06:22:49.226307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:49.226272 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz"] Oct 30 06:22:49.226827 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:49.226489 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" containerID="cri-o://8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae" gracePeriod=30 Oct 30 06:22:50.279595 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:50.279562 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8"] Oct 30 06:22:51.073597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:51.073568 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:51.073779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:51.073625 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:52.694769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.694744 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:22:52.747338 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.747312 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.779993 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780145 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780192 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780263 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780295 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780346 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780425 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780538 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.781512 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.780610 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token\") pod \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\" (UID: \"9e3ba20b-581d-4da1-8e15-ec98b13f08de\") " Oct 30 06:22:52.784012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.783840 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:52.785611 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.785481 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.785711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.785633 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.785888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.785842 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.786200 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786001 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786319 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786724 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786755 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786770 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786783 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786797 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.787824 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.786812 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.794232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.788571 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.794232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.791667 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.794398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.794367 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl" pod "9e3ba20b-581d-4da1-8e15-ec98b13f08de" (UID: "9e3ba20b-581d-4da1-8e15-ec98b13f08de"). InnerVolumeSpecName "kube-api-access-zvnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.828238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.828211 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6"] Oct 30 06:22:52.849090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.849054 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl"] Oct 30 06:22:52.887584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887553 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.887790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887620 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.887790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887663 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.887790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887733 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.887983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887827 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.887983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887888 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.887983 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887956 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.888173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.887910 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:52.888173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888012 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.888173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888125 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9\") pod \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\" (UID: \"4018a797-c102-4cbb-9e8d-8ff7a87098d6\") " Oct 30 06:22:52.888404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888377 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.888590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888563 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.888830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888777 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.888917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888838 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.888917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888854 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnrl\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-kube-api-access-zvnrl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.888917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888869 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.888917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888883 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3ba20b-581d-4da1-8e15-ec98b13f08de-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.888917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.888894 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3ba20b-581d-4da1-8e15-ec98b13f08de-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.889292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.889269 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.889358 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.889336 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.891013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.890989 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.891291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.891265 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.891426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.891409 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:52.891578 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.891561 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9" pod "4018a797-c102-4cbb-9e8d-8ff7a87098d6" (UID: "4018a797-c102-4cbb-9e8d-8ff7a87098d6"). InnerVolumeSpecName "kube-api-access-7jzc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.942252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.942095 5980 generic.go:356] "Generic (PLEG): container finished" podID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerID="e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1" exitCode=0 Oct 30 06:22:52.942252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.942180 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" event={"ID":"4018a797-c102-4cbb-9e8d-8ff7a87098d6","Type":"ContainerDied","Data":"e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1"} Oct 30 06:22:52.942481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.942280 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" event={"ID":"4018a797-c102-4cbb-9e8d-8ff7a87098d6","Type":"ContainerDied","Data":"f91e2064a312080fedd7fff94a9cd7fa3ac76e7594e6d492e731d3f4bf9c1148"} Oct 30 06:22:52.942481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.942333 5980 scope.go:122] "RemoveContainer" containerID="e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1" Oct 30 06:22:52.943326 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.943306 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j" Oct 30 06:22:52.946730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.946703 5980 generic.go:356] "Generic (PLEG): container finished" podID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerID="c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c" exitCode=0 Oct 30 06:22:52.946835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.946768 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" event={"ID":"9e3ba20b-581d-4da1-8e15-ec98b13f08de","Type":"ContainerDied","Data":"c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c"} Oct 30 06:22:52.946835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.946813 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" event={"ID":"9e3ba20b-581d-4da1-8e15-ec98b13f08de","Type":"ContainerDied","Data":"488e06aedbb121ef92942ea7f96a14b6f9cdbc44b1d3ae0916adc50ff0d304ce"} Oct 30 06:22:52.946964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.946898 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d" Oct 30 06:22:52.964760 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.964738 5980 scope.go:122] "RemoveContainer" containerID="e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1" Oct 30 06:22:52.965198 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:52.965083 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1\": container with ID starting with e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1 not found: ID does not exist" containerID="e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1" Oct 30 06:22:52.965198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.965143 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1"} err="failed to get container status \"e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1\": rpc error: code = NotFound desc = could not find container \"e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1\": container with ID starting with e2f3fe0885bb83bd00ed3dbd7cc9e157358732c93f061cba8958812566eb31f1 not found: ID does not exist" Oct 30 06:22:52.965198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.965171 5980 scope.go:122] "RemoveContainer" containerID="c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c" Oct 30 06:22:52.978961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.978942 5980 scope.go:122] "RemoveContainer" containerID="c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c" Oct 30 06:22:52.979574 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:52.979552 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c\": container with ID starting with c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c not found: ID does not exist" containerID="c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c" Oct 30 06:22:52.979742 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.979717 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c"} err="failed to get container status \"c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c\": rpc error: code = NotFound desc = could not find container \"c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c\": container with ID starting with c3d8ea40114b7053dc49623068441623b4993ef433596510e5e183e90238758c not found: ID does not exist" Oct 30 06:22:52.989946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.989922 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.989946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.989947 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.990149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.989956 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.990149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.989965 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jzc9\" (UniqueName: \"kubernetes.io/projected/4018a797-c102-4cbb-9e8d-8ff7a87098d6-kube-api-access-7jzc9\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.990149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.989974 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.990149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:52.989982 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4018a797-c102-4cbb-9e8d-8ff7a87098d6-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.007976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.007956 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j"] Oct 30 06:22:53.011925 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.011902 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k5q6j"] Oct 30 06:22:53.049384 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.048489 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d"] Oct 30 06:22:53.082118 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.082062 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-j2q9d"] Oct 30 06:22:53.590399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.590367 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 06:22:53.701273 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701234 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701315 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701360 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701388 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701455 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701557 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701589 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701651 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701678 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:53.701753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701687 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs\") pod \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\" (UID: \"a274fed5-ff91-44aa-a19f-129c6b9c8b32\") " Oct 30 06:22:53.702267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.701768 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:53.702267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.702140 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:53.702381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.702267 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.702381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.702286 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.702381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.702300 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.702381 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.702352 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:53.702823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.702793 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:53.704416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.704389 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "kube-api-access-jwg9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:53.704416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.704404 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:53.704859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.704839 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:53.705597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.704978 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token" pod "a274fed5-ff91-44aa-a19f-129c6b9c8b32" (UID: "a274fed5-ff91-44aa-a19f-129c6b9c8b32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:53.803340 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.803311 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.803340 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.803339 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.803541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.803353 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.803541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.803363 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.803541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.803372 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a274fed5-ff91-44aa-a19f-129c6b9c8b32-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.803541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.803380 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwg9n\" (UniqueName: \"kubernetes.io/projected/a274fed5-ff91-44aa-a19f-129c6b9c8b32-kube-api-access-jwg9n\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.956368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.956332 5980 generic.go:356] "Generic (PLEG): container finished" podID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerID="16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0" exitCode=0 Oct 30 06:22:53.956561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.956381 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" Oct 30 06:22:53.956561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.956413 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" event={"ID":"a274fed5-ff91-44aa-a19f-129c6b9c8b32","Type":"ContainerDied","Data":"16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0"} Oct 30 06:22:53.956561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.956470 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8" event={"ID":"a274fed5-ff91-44aa-a19f-129c6b9c8b32","Type":"ContainerDied","Data":"87bc74a51464a9184ccbc8f352836fc8675648a5850395f877ec5e2d670b119d"} Oct 30 06:22:53.956561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.956499 5980 scope.go:122] "RemoveContainer" containerID="16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0" Oct 30 06:22:53.968770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.968751 5980 scope.go:122] "RemoveContainer" containerID="6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2" Oct 30 06:22:53.979089 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.979070 5980 scope.go:122] "RemoveContainer" containerID="16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0" Oct 30 06:22:53.979466 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:53.979443 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0\": container with ID starting with 16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0 not found: ID does not exist" containerID="16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0" Oct 30 06:22:53.979560 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.979475 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0"} err="failed to get container status \"16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0\": rpc error: code = NotFound desc = could not find container \"16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0\": container with ID starting with 16a04cef44b3c3ae0fa32d5305e84ca8fc2186ece32a67f92c221a793f6693d0 not found: ID does not exist" Oct 30 06:22:53.979560 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.979492 5980 scope.go:122] "RemoveContainer" containerID="6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2" Oct 30 06:22:53.979803 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:53.979777 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2\": container with ID starting with 6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2 not found: ID does not exist" containerID="6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2" Oct 30 06:22:53.979901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:53.979808 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2"} err="failed to get container status \"6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2\": rpc error: code = NotFound desc = could not find container \"6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2\": container with ID starting with 6385fabdfe84ea63d725a22bbd608f6cc07a71aa22ee8d0a156bd966c6bb10f2 not found: ID does not exist" Oct 30 06:22:54.015694 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.015667 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" path="/var/lib/kubelet/pods/4018a797-c102-4cbb-9e8d-8ff7a87098d6/volumes" Oct 30 06:22:54.016186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.016161 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" path="/var/lib/kubelet/pods/9e3ba20b-581d-4da1-8e15-ec98b13f08de/volumes" Oct 30 06:22:54.016471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.016458 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8"] Oct 30 06:22:54.019750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.019733 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-6mvl8"] Oct 30 06:22:54.506679 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.506610 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 06:22:54.614466 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614433 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.614640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614505 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.614640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614546 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.614640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614621 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.614822 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614751 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.614883 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614858 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:54.614948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614933 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.615012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614969 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.615012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.614971 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:54.615163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615014 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.615326 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615301 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:54.615457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615326 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:54.615457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615052 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data\") pod \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\" (UID: \"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4\") " Oct 30 06:22:54.615945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615927 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.616030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615946 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.616030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615961 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.616030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.615971 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.616232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.616174 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:54.616920 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.616895 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:54.617437 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.617423 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "kube-api-access-r74cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:54.617546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.617527 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:54.617607 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.617553 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo" pod "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" (UID: "f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:54.716936 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.716903 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.716936 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.716931 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74cx\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-kube-api-access-r74cx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.716936 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.716941 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.717434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.716949 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.717434 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.716959 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:22:54.965265 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.965229 5980 generic.go:356] "Generic (PLEG): container finished" podID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerID="8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae" exitCode=0 Oct 30 06:22:54.965479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.965284 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" Oct 30 06:22:54.965479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.965322 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" event={"ID":"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4","Type":"ContainerDied","Data":"8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae"} Oct 30 06:22:54.965479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.965381 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz" event={"ID":"f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4","Type":"ContainerDied","Data":"fddc8784682c2b748ccdcf0873ffd680a1092ed7d76f13b143c50b49aa2c7313"} Oct 30 06:22:54.965479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.965411 5980 scope.go:122] "RemoveContainer" containerID="8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae" Oct 30 06:22:54.977777 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.977750 5980 scope.go:122] "RemoveContainer" containerID="8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae" Oct 30 06:22:54.978101 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:22:54.978083 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae\": container with ID starting with 8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae not found: ID does not exist" containerID="8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae" Oct 30 06:22:54.978205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:54.978144 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae"} err="failed to get container status \"8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae\": rpc error: code = NotFound desc = could not find container \"8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae\": container with ID starting with 8e50fda74d74dea8b8a7ca747a409e86382c7a470fcaafd4213a91f21dc8d8ae not found: ID does not exist" Oct 30 06:22:55.023224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:55.023196 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz"] Oct 30 06:22:55.036740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:55.036713 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-ndhsz"] Oct 30 06:22:56.014310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:56.014279 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" path="/var/lib/kubelet/pods/a274fed5-ff91-44aa-a19f-129c6b9c8b32/volumes" Oct 30 06:22:56.015175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:56.014821 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" path="/var/lib/kubelet/pods/f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4/volumes" Oct 30 06:22:57.829194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:57.829152 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerName="payload-processing" containerID="cri-o://71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2" gracePeriod=25 Oct 30 06:22:58.547268 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:22:58.547239 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr"] Oct 30 06:23:00.043055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:00.043019 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:03.548310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:03.548251 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" containerName="payload-processing" containerID="cri-o://d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837" gracePeriod=25 Oct 30 06:23:05.301746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.301688 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" podUID="48abb063-1fc8-435b-af07-9fd4c43ed944" containerName="main" containerID="cri-o://20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0" gracePeriod=45 Oct 30 06:23:05.582805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.582779 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:23:05.676724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.676691 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home\") pod \"48abb063-1fc8-435b-af07-9fd4c43ed944\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " Oct 30 06:23:05.677013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.676996 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs\" (UniqueName: \"kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs\") pod \"48abb063-1fc8-435b-af07-9fd4c43ed944\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " Oct 30 06:23:05.677753 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.677729 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm\") pod \"48abb063-1fc8-435b-af07-9fd4c43ed944\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " Oct 30 06:23:05.677885 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.677781 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache\") pod \"48abb063-1fc8-435b-af07-9fd4c43ed944\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " Oct 30 06:23:05.677944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.677895 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26\" (UniqueName: \"kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26\") pod \"48abb063-1fc8-435b-af07-9fd4c43ed944\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " Oct 30 06:23:05.678303 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.678060 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir\") pod \"48abb063-1fc8-435b-af07-9fd4c43ed944\" (UID: \"48abb063-1fc8-435b-af07-9fd4c43ed944\") " Oct 30 06:23:05.678457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.678396 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir" pod "48abb063-1fc8-435b-af07-9fd4c43ed944" (UID: "48abb063-1fc8-435b-af07-9fd4c43ed944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.678457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.678172 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home" pod "48abb063-1fc8-435b-af07-9fd4c43ed944" (UID: "48abb063-1fc8-435b-af07-9fd4c43ed944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.678582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.678513 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache" pod "48abb063-1fc8-435b-af07-9fd4c43ed944" (UID: "48abb063-1fc8-435b-af07-9fd4c43ed944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.679162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.679140 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.679429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.679165 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.679429 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.679177 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.681856 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.681819 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm" pod "48abb063-1fc8-435b-af07-9fd4c43ed944" (UID: "48abb063-1fc8-435b-af07-9fd4c43ed944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:05.681945 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.681881 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26" pod "48abb063-1fc8-435b-af07-9fd4c43ed944" (UID: "48abb063-1fc8-435b-af07-9fd4c43ed944"). InnerVolumeSpecName "kube-api-access-cjw26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:05.682041 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.682022 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs" pod "48abb063-1fc8-435b-af07-9fd4c43ed944" (UID: "48abb063-1fc8-435b-af07-9fd4c43ed944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:05.780860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.780825 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48abb063-1fc8-435b-af07-9fd4c43ed944-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.780860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.780859 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48abb063-1fc8-435b-af07-9fd4c43ed944-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.781080 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:05.780874 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjw26\" (UniqueName: \"kubernetes.io/projected/48abb063-1fc8-435b-af07-9fd4c43ed944-kube-api-access-cjw26\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:06.103171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.096555 5980 generic.go:356] "Generic (PLEG): container finished" podID="48abb063-1fc8-435b-af07-9fd4c43ed944" containerID="20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0" exitCode=0 Oct 30 06:23:06.103171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.096667 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" Oct 30 06:23:06.103171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.096858 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" event={"ID":"48abb063-1fc8-435b-af07-9fd4c43ed944","Type":"ContainerDied","Data":"20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0"} Oct 30 06:23:06.103171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.096906 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" event={"ID":"48abb063-1fc8-435b-af07-9fd4c43ed944","Type":"ContainerDied","Data":"35c0a9e1c1b9420df4b2d1b7fb1aa97586400610df37b1a0a2ddad6220df023d"} Oct 30 06:23:06.103171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.096931 5980 scope.go:122] "RemoveContainer" containerID="20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0" Oct 30 06:23:06.114958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.114933 5980 scope.go:122] "RemoveContainer" containerID="20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0" Oct 30 06:23:06.115356 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:06.115333 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0\": container with ID starting with 20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0 not found: ID does not exist" containerID="20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0" Oct 30 06:23:06.115479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.115365 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0"} err="failed to get container status \"20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0\": rpc error: code = NotFound desc = could not find container \"20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0\": container with ID starting with 20a73aa4af3b8afef8239091265cb0306f70ecf4d5495e93b5ad597adf98d5e0 not found: ID does not exist" Oct 30 06:23:06.154222 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.154191 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8"] Oct 30 06:23:06.167700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.167672 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8"] Oct 30 06:23:06.526264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:06.526161 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-bf87e1/ipp-route-bf87e1-kserve-59d894569c-4mdz8" podUID="48abb063-1fc8-435b-af07-9fd4c43ed944" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/ready\": context deadline exceeded" Oct 30 06:23:07.872281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:07.872235 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" containerID="cri-o://297a0d17cc057bf1ce303380039aedc895024c3a9a0d8bf4739c954859f93634" gracePeriod=45 Oct 30 06:23:07.947431 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:07.947404 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q"] Oct 30 06:23:07.947670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:07.947631 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" podUID="5fce3f95-3101-4a0b-9bed-1b4fafef341c" containerName="maas-api" containerID="cri-o://7b7c6e630d33aaa0da0366934a136b42670d21ee339f55111287ceedc3c0f445" gracePeriod=30 Oct 30 06:23:08.016644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.016612 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48abb063-1fc8-435b-af07-9fd4c43ed944" path="/var/lib/kubelet/pods/48abb063-1fc8-435b-af07-9fd4c43ed944/volumes" Oct 30 06:23:08.027126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.027084 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:08.113970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.113930 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 30 06:23:08.117169 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.117139 5980 generic.go:356] "Generic (PLEG): container finished" podID="bb495806-9f5c-47d5-b035-f6b546928092" containerID="297a0d17cc057bf1ce303380039aedc895024c3a9a0d8bf4739c954859f93634" exitCode=0 Oct 30 06:23:08.117311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.117258 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" event={"ID":"bb495806-9f5c-47d5-b035-f6b546928092","Type":"ContainerDied","Data":"297a0d17cc057bf1ce303380039aedc895024c3a9a0d8bf4739c954859f93634"} Oct 30 06:23:08.119022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.119000 5980 generic.go:356] "Generic (PLEG): container finished" podID="5fce3f95-3101-4a0b-9bed-1b4fafef341c" containerID="7b7c6e630d33aaa0da0366934a136b42670d21ee339f55111287ceedc3c0f445" exitCode=0 Oct 30 06:23:08.119177 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.119159 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" event={"ID":"5fce3f95-3101-4a0b-9bed-1b4fafef341c","Type":"ContainerDied","Data":"7b7c6e630d33aaa0da0366934a136b42670d21ee339f55111287ceedc3c0f445"} Oct 30 06:23:08.288629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.288417 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 06:23:08.321339 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.321315 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 06:23:08.331564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.331538 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs\") pod \"bb495806-9f5c-47d5-b035-f6b546928092\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " Oct 30 06:23:08.331700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.331604 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home\") pod \"bb495806-9f5c-47d5-b035-f6b546928092\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " Oct 30 06:23:08.331700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.331647 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir\") pod \"bb495806-9f5c-47d5-b035-f6b546928092\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " Oct 30 06:23:08.331862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.331843 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58\" (UniqueName: \"kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58\") pod \"bb495806-9f5c-47d5-b035-f6b546928092\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " Oct 30 06:23:08.331915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.331901 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache\") pod \"bb495806-9f5c-47d5-b035-f6b546928092\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " Oct 30 06:23:08.332122 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332089 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm\") pod \"bb495806-9f5c-47d5-b035-f6b546928092\" (UID: \"bb495806-9f5c-47d5-b035-f6b546928092\") " Oct 30 06:23:08.332202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332176 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home" pod "bb495806-9f5c-47d5-b035-f6b546928092" (UID: "bb495806-9f5c-47d5-b035-f6b546928092"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.332259 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332202 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir" pod "bb495806-9f5c-47d5-b035-f6b546928092" (UID: "bb495806-9f5c-47d5-b035-f6b546928092"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.332835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332538 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache" pod "bb495806-9f5c-47d5-b035-f6b546928092" (UID: "bb495806-9f5c-47d5-b035-f6b546928092"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.332835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332810 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.332835 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332831 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.333047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.332843 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.335590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.335544 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs" pod "bb495806-9f5c-47d5-b035-f6b546928092" (UID: "bb495806-9f5c-47d5-b035-f6b546928092"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:08.335590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.335552 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm" pod "bb495806-9f5c-47d5-b035-f6b546928092" (UID: "bb495806-9f5c-47d5-b035-f6b546928092"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.335590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.335572 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58" pod "bb495806-9f5c-47d5-b035-f6b546928092" (UID: "bb495806-9f5c-47d5-b035-f6b546928092"). InnerVolumeSpecName "kube-api-access-rpm58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:08.433999 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.433898 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs\") pod \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " Oct 30 06:23:08.434228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434056 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca\") pod \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " Oct 30 06:23:08.434228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434211 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\") pod \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " Oct 30 06:23:08.434374 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434334 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc\" (UniqueName: \"kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc\") pod \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\" (UID: \"5fce3f95-3101-4a0b-9bed-1b4fafef341c\") " Oct 30 06:23:08.434830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434654 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca" pod "5fce3f95-3101-4a0b-9bed-1b4fafef341c" (UID: "5fce3f95-3101-4a0b-9bed-1b4fafef341c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:08.434942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434892 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpm58\" (UniqueName: \"kubernetes.io/projected/bb495806-9f5c-47d5-b035-f6b546928092-kube-api-access-rpm58\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.434942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434912 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb495806-9f5c-47d5-b035-f6b546928092-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.434942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434926 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb495806-9f5c-47d5-b035-f6b546928092-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.434942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.434940 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fce3f95-3101-4a0b-9bed-1b4fafef341c-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.437058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.437034 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs" pod "5fce3f95-3101-4a0b-9bed-1b4fafef341c" (UID: "5fce3f95-3101-4a0b-9bed-1b4fafef341c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:08.437337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.437246 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc" pod "5fce3f95-3101-4a0b-9bed-1b4fafef341c" (UID: "5fce3f95-3101-4a0b-9bed-1b4fafef341c"). InnerVolumeSpecName "kube-api-access-bv7mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:08.437529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.437509 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls" pod "5fce3f95-3101-4a0b-9bed-1b4fafef341c" (UID: "5fce3f95-3101-4a0b-9bed-1b4fafef341c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:08.535977 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.535946 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.535977 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.535973 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv7mc\" (UniqueName: \"kubernetes.io/projected/5fce3f95-3101-4a0b-9bed-1b4fafef341c-kube-api-access-bv7mc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.535977 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:08.535983 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fce3f95-3101-4a0b-9bed-1b4fafef341c-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:09.126083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.126055 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" Oct 30 06:23:09.126083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.126075 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl" event={"ID":"bb495806-9f5c-47d5-b035-f6b546928092","Type":"ContainerDied","Data":"32b116714931fb20f87e24561a734e78b4afc4519e2fb2c8633eba684ce205d5"} Oct 30 06:23:09.126672 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.126151 5980 scope.go:122] "RemoveContainer" containerID="297a0d17cc057bf1ce303380039aedc895024c3a9a0d8bf4739c954859f93634" Oct 30 06:23:09.127622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.127602 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" event={"ID":"5fce3f95-3101-4a0b-9bed-1b4fafef341c","Type":"ContainerDied","Data":"27af4c8d62d6501c918505acc8479acbb4405bdb6c2692fae979ed0b9402027f"} Oct 30 06:23:09.127685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.127629 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q" Oct 30 06:23:09.137841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.137821 5980 scope.go:122] "RemoveContainer" containerID="7b7c6e630d33aaa0da0366934a136b42670d21ee339f55111287ceedc3c0f445" Oct 30 06:23:09.181831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.181800 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl"] Oct 30 06:23:09.195126 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.195088 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-w1-4a68a1-kserve-6d94b965b7-pmlsl"] Oct 30 06:23:09.225061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.225038 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q"] Oct 30 06:23:09.228723 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:09.228703 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-c73fcb-885c74b49-l5p8q"] Oct 30 06:23:10.014376 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:10.014344 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fce3f95-3101-4a0b-9bed-1b4fafef341c" path="/var/lib/kubelet/pods/5fce3f95-3101-4a0b-9bed-1b4fafef341c/volumes" Oct 30 06:23:10.014811 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:10.014797 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb495806-9f5c-47d5-b035-f6b546928092" path="/var/lib/kubelet/pods/bb495806-9f5c-47d5-b035-f6b546928092/volumes" Oct 30 06:23:10.043582 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:10.043554 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:10.848841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:10.848808 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d"] Oct 30 06:23:13.088575 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:13.088540 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:23:15.849789 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:15.849753 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" podUID="a14d1d94-d361-4634-9cbc-cf4621f87cc1" containerName="payload-pre-processing" containerID="cri-o://c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d" gracePeriod=25 Oct 30 06:23:16.129728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.129699 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:23:16.176925 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.176890 5980 generic.go:356] "Generic (PLEG): container finished" podID="a14d1d94-d361-4634-9cbc-cf4621f87cc1" containerID="c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d" exitCode=0 Oct 30 06:23:16.178011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.177222 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" Oct 30 06:23:16.178011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.177261 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" event={"ID":"a14d1d94-d361-4634-9cbc-cf4621f87cc1","Type":"ContainerDied","Data":"c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d"} Oct 30 06:23:16.178011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.177881 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d" event={"ID":"a14d1d94-d361-4634-9cbc-cf4621f87cc1","Type":"ContainerDied","Data":"f1f8a5a5fc61094bfe27d33a0f203254c6677d86f93b662417cac36899521974"} Oct 30 06:23:16.178011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.177906 5980 scope.go:122] "RemoveContainer" containerID="c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d" Oct 30 06:23:16.191376 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.191349 5980 scope.go:122] "RemoveContainer" containerID="c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d" Oct 30 06:23:16.191946 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:16.191710 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d\": container with ID starting with c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d not found: ID does not exist" containerID="c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d" Oct 30 06:23:16.191946 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.191746 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d"} err="failed to get container status \"c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d\": rpc error: code = NotFound desc = could not find container \"c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d\": container with ID starting with c93f7cba0dd9728cbb439c6fcb3e7a45363ef871d45992a15f957d647d22448d not found: ID does not exist" Oct 30 06:23:16.255163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.252554 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw\" (UniqueName: \"kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw\") pod \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " Oct 30 06:23:16.255163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.252780 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume\") pod \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\" (UID: \"a14d1d94-d361-4634-9cbc-cf4621f87cc1\") " Oct 30 06:23:16.257153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.257098 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume" pod "a14d1d94-d361-4634-9cbc-cf4621f87cc1" (UID: "a14d1d94-d361-4634-9cbc-cf4621f87cc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:16.259892 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.259862 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw" pod "a14d1d94-d361-4634-9cbc-cf4621f87cc1" (UID: "a14d1d94-d361-4634-9cbc-cf4621f87cc1"). InnerVolumeSpecName "kube-api-access-66qxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:16.354074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.354002 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a14d1d94-d361-4634-9cbc-cf4621f87cc1-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:16.354074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.354029 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66qxw\" (UniqueName: \"kubernetes.io/projected/a14d1d94-d361-4634-9cbc-cf4621f87cc1-kube-api-access-66qxw\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:16.533118 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.533070 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d"] Oct 30 06:23:16.540540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:16.540514 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-fb489d94c-c958d"] Oct 30 06:23:18.021796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.021750 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14d1d94-d361-4634-9cbc-cf4621f87cc1" path="/var/lib/kubelet/pods/a14d1d94-d361-4634-9cbc-cf4621f87cc1/volumes" Oct 30 06:23:18.028494 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.028456 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:18.106060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.106023 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 06:23:18.194728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.194699 5980 generic.go:356] "Generic (PLEG): container finished" podID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerID="71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2" exitCode=0 Oct 30 06:23:18.194935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.194757 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" Oct 30 06:23:18.194935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.194780 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" event={"ID":"fda4eb6a-85f1-428e-8193-33da6f1482e4","Type":"ContainerDied","Data":"71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2"} Oct 30 06:23:18.194935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.194836 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6" event={"ID":"fda4eb6a-85f1-428e-8193-33da6f1482e4","Type":"ContainerDied","Data":"a71a5c0ba3a64774570443671014654babf5bc985430a96838ad163ce57010c2"} Oct 30 06:23:18.194935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.194862 5980 scope.go:122] "RemoveContainer" containerID="71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2" Oct 30 06:23:18.214488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.214468 5980 scope.go:122] "RemoveContainer" containerID="71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2" Oct 30 06:23:18.214806 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:18.214784 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2\": container with ID starting with 71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2 not found: ID does not exist" containerID="71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2" Oct 30 06:23:18.214906 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.214813 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2"} err="failed to get container status \"71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2\": rpc error: code = NotFound desc = could not find container \"71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2\": container with ID starting with 71c7c082e1523f9375efb7a1d1fa139587de5f42adf5fd7c7cd2ae2d93b367f2 not found: ID does not exist" Oct 30 06:23:18.284324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.284298 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume\") pod \"fda4eb6a-85f1-428e-8193-33da6f1482e4\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " Oct 30 06:23:18.284460 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.284357 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj\" (UniqueName: \"kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj\") pod \"fda4eb6a-85f1-428e-8193-33da6f1482e4\" (UID: \"fda4eb6a-85f1-428e-8193-33da6f1482e4\") " Oct 30 06:23:18.284671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.284651 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume" pod "fda4eb6a-85f1-428e-8193-33da6f1482e4" (UID: "fda4eb6a-85f1-428e-8193-33da6f1482e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:18.284948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.284843 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda4eb6a-85f1-428e-8193-33da6f1482e4-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.286730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.286695 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj" pod "fda4eb6a-85f1-428e-8193-33da6f1482e4" (UID: "fda4eb6a-85f1-428e-8193-33da6f1482e4"). InnerVolumeSpecName "kube-api-access-b75fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:18.386382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.386313 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75fj\" (UniqueName: \"kubernetes.io/projected/fda4eb6a-85f1-428e-8193-33da6f1482e4-kube-api-access-b75fj\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.543728 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.543694 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6"] Oct 30 06:23:18.556833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:18.556801 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-1354a5-5b6cc9875c-688l6"] Oct 30 06:23:20.013809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:20.013778 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" path="/var/lib/kubelet/pods/fda4eb6a-85f1-428e-8193-33da6f1482e4/volumes" Oct 30 06:23:21.972045 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:21.972004 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf"] Oct 30 06:23:21.979871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:21.979844 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t"] Oct 30 06:23:22.005779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:22.005750 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr"] Oct 30 06:23:23.821168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.821148 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 06:23:23.852779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.852750 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl\" (UniqueName: \"kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl\") pod \"4d18805d-84cd-4b64-b910-0c0b14325230\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " Oct 30 06:23:23.852966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.852950 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume\") pod \"4d18805d-84cd-4b64-b910-0c0b14325230\" (UID: \"4d18805d-84cd-4b64-b910-0c0b14325230\") " Oct 30 06:23:23.853320 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.853298 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume" pod "4d18805d-84cd-4b64-b910-0c0b14325230" (UID: "4d18805d-84cd-4b64-b910-0c0b14325230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:23.853417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.853397 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d18805d-84cd-4b64-b910-0c0b14325230-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:23.854908 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.854873 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl" pod "4d18805d-84cd-4b64-b910-0c0b14325230" (UID: "4d18805d-84cd-4b64-b910-0c0b14325230"). InnerVolumeSpecName "kube-api-access-cp4wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:23.954800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:23.954760 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4wl\" (UniqueName: \"kubernetes.io/projected/4d18805d-84cd-4b64-b910-0c0b14325230-kube-api-access-cp4wl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.249637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.248215 5980 generic.go:356] "Generic (PLEG): container finished" podID="4d18805d-84cd-4b64-b910-0c0b14325230" containerID="d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837" exitCode=0 Oct 30 06:23:24.249637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.248351 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" event={"ID":"4d18805d-84cd-4b64-b910-0c0b14325230","Type":"ContainerDied","Data":"d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837"} Oct 30 06:23:24.249637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.248400 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" event={"ID":"4d18805d-84cd-4b64-b910-0c0b14325230","Type":"ContainerDied","Data":"f53d3f755d61b68d36ebc13aa7739ceaf8624f313fa85724bf0112a72ae227bb"} Oct 30 06:23:24.249637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.248431 5980 scope.go:122] "RemoveContainer" containerID="d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837" Oct 30 06:23:24.249637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.248582 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr" Oct 30 06:23:24.261440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.261420 5980 scope.go:122] "RemoveContainer" containerID="d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837" Oct 30 06:23:24.261740 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:24.261720 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837\": container with ID starting with d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837 not found: ID does not exist" containerID="d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837" Oct 30 06:23:24.261800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.261750 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837"} err="failed to get container status \"d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837\": rpc error: code = NotFound desc = could not find container \"d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837\": container with ID starting with d6d11007b28bb021aae711b843f064eb10632df329b7dfa9dca5869d869ba837 not found: ID does not exist" Oct 30 06:23:24.308282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.308251 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr"] Oct 30 06:23:24.311580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:24.311554 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4a68a1-7fc787db87-z5rwr"] Oct 30 06:23:26.023443 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:26.023411 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" path="/var/lib/kubelet/pods/4d18805d-84cd-4b64-b910-0c0b14325230/volumes" Oct 30 06:23:27.189131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.187076 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl"] Oct 30 06:23:27.189131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.187335 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" podUID="f1fc41cf-f675-4ada-b506-def7d09a8f41" containerName="maas-api" containerID="cri-o://672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6" gracePeriod=30 Oct 30 06:23:27.537656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.537631 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 06:23:27.660677 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.660646 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs\") pod \"f1fc41cf-f675-4ada-b506-def7d09a8f41\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " Oct 30 06:23:27.660878 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.660774 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg\" (UniqueName: \"kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg\") pod \"f1fc41cf-f675-4ada-b506-def7d09a8f41\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " Oct 30 06:23:27.660963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.660880 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\") pod \"f1fc41cf-f675-4ada-b506-def7d09a8f41\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " Oct 30 06:23:27.661021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.660957 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca\") pod \"f1fc41cf-f675-4ada-b506-def7d09a8f41\" (UID: \"f1fc41cf-f675-4ada-b506-def7d09a8f41\") " Oct 30 06:23:27.661395 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.661367 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca" pod "f1fc41cf-f675-4ada-b506-def7d09a8f41" (UID: "f1fc41cf-f675-4ada-b506-def7d09a8f41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:27.661569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.661548 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc41cf-f675-4ada-b506-def7d09a8f41-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.662917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.662894 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs" pod "f1fc41cf-f675-4ada-b506-def7d09a8f41" (UID: "f1fc41cf-f675-4ada-b506-def7d09a8f41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:27.663305 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.663285 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls" pod "f1fc41cf-f675-4ada-b506-def7d09a8f41" (UID: "f1fc41cf-f675-4ada-b506-def7d09a8f41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:27.663522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.663504 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg" pod "f1fc41cf-f675-4ada-b506-def7d09a8f41" (UID: "f1fc41cf-f675-4ada-b506-def7d09a8f41"). InnerVolumeSpecName "kube-api-access-5qprg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:27.775190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.775071 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.775190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.775127 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fc41cf-f675-4ada-b506-def7d09a8f41-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:27.775190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:27.775143 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qprg\" (UniqueName: \"kubernetes.io/projected/f1fc41cf-f675-4ada-b506-def7d09a8f41-kube-api-access-5qprg\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:28.281176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.279461 5980 generic.go:356] "Generic (PLEG): container finished" podID="f1fc41cf-f675-4ada-b506-def7d09a8f41" containerID="672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6" exitCode=0 Oct 30 06:23:28.281176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.279524 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" Oct 30 06:23:28.281176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.279517 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" event={"ID":"f1fc41cf-f675-4ada-b506-def7d09a8f41","Type":"ContainerDied","Data":"672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6"} Oct 30 06:23:28.281176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.279663 5980 scope.go:122] "RemoveContainer" containerID="672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6" Oct 30 06:23:28.281176 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.280326 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl" event={"ID":"f1fc41cf-f675-4ada-b506-def7d09a8f41","Type":"ContainerDied","Data":"6d51d2bf2aa7b2552c62d745e28a9b3dbec2e053779fdad4df636168331277ca"} Oct 30 06:23:28.292594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.292574 5980 scope.go:122] "RemoveContainer" containerID="672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6" Oct 30 06:23:28.293280 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:28.293256 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6\": container with ID starting with 672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6 not found: ID does not exist" containerID="672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6" Oct 30 06:23:28.293370 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.293289 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6"} err="failed to get container status \"672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6\": rpc error: code = NotFound desc = could not find container \"672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6\": container with ID starting with 672c5072fd3a6f30d3df80a842e648b91559e965592915b19dae770c82231af6 not found: ID does not exist" Oct 30 06:23:28.345258 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.345228 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl"] Oct 30 06:23:28.352866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:28.352841 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-1354a5-6d4fc55f9c-rxxwl"] Oct 30 06:23:30.014617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:30.014580 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fc41cf-f675-4ada-b506-def7d09a8f41" path="/var/lib/kubelet/pods/f1fc41cf-f675-4ada-b506-def7d09a8f41/volumes" Oct 30 06:23:36.512911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:36.512818 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp"] Oct 30 06:23:36.513567 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:36.513127 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" containerID="cri-o://a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b" gracePeriod=30 Oct 30 06:23:37.002072 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.002021 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" containerID="cri-o://e4fabcd1e0192d61002fc526b9962a34ffb6faaa21af1bc4be6f5976cec7ce8b" gracePeriod=45 Oct 30 06:23:37.005643 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.005588 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" containerID="cri-o://cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459" gracePeriod=45 Oct 30 06:23:37.027292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.027258 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" containerID="cri-o://0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b" gracePeriod=45 Oct 30 06:23:37.326074 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.326046 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 06:23:37.330165 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.330145 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 06:23:37.345664 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.345640 5980 generic.go:356] "Generic (PLEG): container finished" podID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerID="e4fabcd1e0192d61002fc526b9962a34ffb6faaa21af1bc4be6f5976cec7ce8b" exitCode=0 Oct 30 06:23:37.345815 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.345723 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" event={"ID":"1c48ddd4-40f4-41ed-9350-5e24c07ef671","Type":"ContainerDied","Data":"e4fabcd1e0192d61002fc526b9962a34ffb6faaa21af1bc4be6f5976cec7ce8b"} Oct 30 06:23:37.347445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.347424 5980 generic.go:356] "Generic (PLEG): container finished" podID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerID="0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b" exitCode=0 Oct 30 06:23:37.347564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.347515 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" Oct 30 06:23:37.347638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.347595 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" event={"ID":"70cbe530-2176-4450-881b-74bd0a41eb1a","Type":"ContainerDied","Data":"0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b"} Oct 30 06:23:37.347638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.347628 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr" event={"ID":"70cbe530-2176-4450-881b-74bd0a41eb1a","Type":"ContainerDied","Data":"f6ace4886d3a0d65baca85f80fecbbb446f0e12884b04026f4894fe3f3da8e51"} Oct 30 06:23:37.347778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.347651 5980 scope.go:122] "RemoveContainer" containerID="0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b" Oct 30 06:23:37.349965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.349857 5980 generic.go:356] "Generic (PLEG): container finished" podID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerID="cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459" exitCode=0 Oct 30 06:23:37.349965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.349901 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" Oct 30 06:23:37.350206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.349902 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" event={"ID":"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f","Type":"ContainerDied","Data":"cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459"} Oct 30 06:23:37.350206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.350015 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t" event={"ID":"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f","Type":"ContainerDied","Data":"9e6889001ee562d1369ef588dc43f401da14e2ffba5ba048ca1db5092d09a1aa"} Oct 30 06:23:37.358852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.358834 5980 scope.go:122] "RemoveContainer" containerID="0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b" Oct 30 06:23:37.359223 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:37.359195 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b\": container with ID starting with 0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b not found: ID does not exist" containerID="0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b" Oct 30 06:23:37.359298 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.359234 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b"} err="failed to get container status \"0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b\": rpc error: code = NotFound desc = could not find container \"0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b\": container with ID starting with 0cd35a9d388a2d1d47df4b4d7f598a7afd975865cdfece69c93c4407a2565d3b not found: ID does not exist" Oct 30 06:23:37.359298 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.359261 5980 scope.go:122] "RemoveContainer" containerID="cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459" Oct 30 06:23:37.370772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.370746 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm\") pod \"70cbe530-2176-4450-881b-74bd0a41eb1a\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " Oct 30 06:23:37.370895 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.370878 5980 scope.go:122] "RemoveContainer" containerID="cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459" Oct 30 06:23:37.371025 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.370880 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs\") pod \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " Oct 30 06:23:37.371476 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.371454 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir\") pod \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " Oct 30 06:23:37.371618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.371508 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home\") pod \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " Oct 30 06:23:37.371618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.371557 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home\") pod \"70cbe530-2176-4450-881b-74bd0a41eb1a\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372277 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home" pod "70cbe530-2176-4450-881b-74bd0a41eb1a" (UID: "70cbe530-2176-4450-881b-74bd0a41eb1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372297 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir" pod "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" (UID: "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372348 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home" pod "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" (UID: "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372430 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache\") pod \"70cbe530-2176-4450-881b-74bd0a41eb1a\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372509 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache\") pod \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372509 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache" pod "70cbe530-2176-4450-881b-74bd0a41eb1a" (UID: "70cbe530-2176-4450-881b-74bd0a41eb1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372542 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn\" (UniqueName: \"kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn\") pod \"70cbe530-2176-4450-881b-74bd0a41eb1a\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372581 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb\" (UniqueName: \"kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb\") pod \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372628 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir\") pod \"70cbe530-2176-4450-881b-74bd0a41eb1a\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372667 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs\" (UniqueName: \"kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs\") pod \"70cbe530-2176-4450-881b-74bd0a41eb1a\" (UID: \"70cbe530-2176-4450-881b-74bd0a41eb1a\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372766 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm\") pod \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\" (UID: \"c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f\") " Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.372816 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache" pod "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" (UID: "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.373383 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.373404 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.373417 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.373429 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.373443 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.374624 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs" pod "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" (UID: "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.375002 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir" pod "70cbe530-2176-4450-881b-74bd0a41eb1a" (UID: "70cbe530-2176-4450-881b-74bd0a41eb1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.375339 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm" pod "70cbe530-2176-4450-881b-74bd0a41eb1a" (UID: "70cbe530-2176-4450-881b-74bd0a41eb1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.377039 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn" pod "70cbe530-2176-4450-881b-74bd0a41eb1a" (UID: "70cbe530-2176-4450-881b-74bd0a41eb1a"). InnerVolumeSpecName "kube-api-access-qv9mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:37.377152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.377092 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb" pod "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" (UID: "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f"). InnerVolumeSpecName "kube-api-access-4gfwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:37.378412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.378012 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs" pod "70cbe530-2176-4450-881b-74bd0a41eb1a" (UID: "70cbe530-2176-4450-881b-74bd0a41eb1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:37.378808 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:37.378779 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459\": container with ID starting with cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459 not found: ID does not exist" containerID="cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459" Oct 30 06:23:37.378939 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.378819 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459"} err="failed to get container status \"cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459\": rpc error: code = NotFound desc = could not find container \"cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459\": container with ID starting with cb0e602fe9449e807d6a77042439b95ead6a6d922951d0fdcff9e6e8803e9459 not found: ID does not exist" Oct 30 06:23:37.380663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.380592 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 06:23:37.384016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.383992 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm" pod "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" (UID: "c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.473923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.473892 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home\") pod \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " Oct 30 06:23:37.474145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.473974 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm\") pod \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " Oct 30 06:23:37.474145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474015 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir\") pod \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " Oct 30 06:23:37.474145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474062 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl\" (UniqueName: \"kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl\") pod \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " Oct 30 06:23:37.474145 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474078 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs\") pod \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " Oct 30 06:23:37.474399 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474162 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache\") pod \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\" (UID: \"1c48ddd4-40f4-41ed-9350-5e24c07ef671\") " Oct 30 06:23:37.474688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474614 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.474688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474637 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.474688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474651 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.474688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474668 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9mn\" (UniqueName: \"kubernetes.io/projected/70cbe530-2176-4450-881b-74bd0a41eb1a-kube-api-access-qv9mn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.474688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474682 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfwb\" (UniqueName: \"kubernetes.io/projected/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f-kube-api-access-4gfwb\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.475022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474697 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70cbe530-2176-4450-881b-74bd0a41eb1a-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.475022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474710 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cbe530-2176-4450-881b-74bd0a41eb1a-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.475022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474964 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home" pod "1c48ddd4-40f4-41ed-9350-5e24c07ef671" (UID: "1c48ddd4-40f4-41ed-9350-5e24c07ef671"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.475022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.474998 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache" pod "1c48ddd4-40f4-41ed-9350-5e24c07ef671" (UID: "1c48ddd4-40f4-41ed-9350-5e24c07ef671"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.475344 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.475313 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir" pod "1c48ddd4-40f4-41ed-9350-5e24c07ef671" (UID: "1c48ddd4-40f4-41ed-9350-5e24c07ef671"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.477904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.477878 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs" pod "1c48ddd4-40f4-41ed-9350-5e24c07ef671" (UID: "1c48ddd4-40f4-41ed-9350-5e24c07ef671"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:37.478028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.477904 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl" pod "1c48ddd4-40f4-41ed-9350-5e24c07ef671" (UID: "1c48ddd4-40f4-41ed-9350-5e24c07ef671"). InnerVolumeSpecName "kube-api-access-kpjxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:37.478146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.478057 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm" pod "1c48ddd4-40f4-41ed-9350-5e24c07ef671" (UID: "1c48ddd4-40f4-41ed-9350-5e24c07ef671"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:37.575467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.575440 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.575467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.575466 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.575467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.575475 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.575961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.575482 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.575961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.575490 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjxl\" (UniqueName: \"kubernetes.io/projected/1c48ddd4-40f4-41ed-9350-5e24c07ef671-kube-api-access-kpjxl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.575961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.575499 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c48ddd4-40f4-41ed-9350-5e24c07ef671-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:37.699969 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.699931 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr"] Oct 30 06:23:37.703904 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.703882 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-128436/simulator-w3-128436-kserve-7758f99944-xf6rr"] Oct 30 06:23:37.740134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.740080 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t"] Oct 30 06:23:37.744371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:37.744351 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-w3-128436-kserve-6c75565476-xjq6t"] Oct 30 06:23:38.013570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.013498 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" path="/var/lib/kubelet/pods/70cbe530-2176-4450-881b-74bd0a41eb1a/volumes" Oct 30 06:23:38.013911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.013898 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" path="/var/lib/kubelet/pods/c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f/volumes" Oct 30 06:23:38.360597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.360569 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" Oct 30 06:23:38.360778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.360569 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf" event={"ID":"1c48ddd4-40f4-41ed-9350-5e24c07ef671","Type":"ContainerDied","Data":"18ae16684f376aabaa4ad693345f7022cbde23f4daae571f31a49daf678b6696"} Oct 30 06:23:38.360778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.360695 5980 scope.go:122] "RemoveContainer" containerID="e4fabcd1e0192d61002fc526b9962a34ffb6faaa21af1bc4be6f5976cec7ce8b" Oct 30 06:23:38.421245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.421211 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf"] Oct 30 06:23:38.434271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:38.434244 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-128436/distinct-2-w3-128436-kserve-5f97768687-wqdrf"] Oct 30 06:23:39.037831 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:39.037800 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz"] Oct 30 06:23:39.038320 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:39.038006 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" containerID="cri-o://731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575" gracePeriod=30 Oct 30 06:23:40.013646 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:40.013604 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" path="/var/lib/kubelet/pods/1c48ddd4-40f4-41ed-9350-5e24c07ef671/volumes" Oct 30 06:23:41.784821 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.784747 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 06:23:41.829433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829400 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829457 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829496 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829574 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829690 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829790 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829739 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829796 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829781 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.829907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829827 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829856 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy\") pod \"f8723f03-8296-4c76-b138-6375e3561d45\" (UID: \"f8723f03-8296-4c76-b138-6375e3561d45\") " Oct 30 06:23:41.829907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829855 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.829907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829866 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:41.830271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.829997 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.830271 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.830017 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.830408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.830379 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.830408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.830402 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.830522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.830411 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8723f03-8296-4c76-b138-6375e3561d45-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.830522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.830423 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.830522 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.830434 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.832533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.832504 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "kube-api-access-zlnvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:41.832659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.832539 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:41.832659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.832642 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.832740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.832723 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo" pod "f8723f03-8296-4c76-b138-6375e3561d45" (UID: "f8723f03-8296-4c76-b138-6375e3561d45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:41.931622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.931581 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.931622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.931613 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8723f03-8296-4c76-b138-6375e3561d45-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.931622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.931622 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8723f03-8296-4c76-b138-6375e3561d45-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.931622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:41.931630 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnvq\" (UniqueName: \"kubernetes.io/projected/f8723f03-8296-4c76-b138-6375e3561d45-kube-api-access-zlnvq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:42.074152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.074091 5980 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:23:42.074335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.074173 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:23:42.420149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.420028 5980 generic.go:356] "Generic (PLEG): container finished" podID="f8723f03-8296-4c76-b138-6375e3561d45" containerID="a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b" exitCode=0 Oct 30 06:23:42.420149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.420082 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" Oct 30 06:23:42.420388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.420151 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" event={"ID":"f8723f03-8296-4c76-b138-6375e3561d45","Type":"ContainerDied","Data":"a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b"} Oct 30 06:23:42.420388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.420208 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp" event={"ID":"f8723f03-8296-4c76-b138-6375e3561d45","Type":"ContainerDied","Data":"dc2329f34b4e63bc3a0280c64cd96c2acc68d3b17235798db1e94f9e23c681bf"} Oct 30 06:23:42.420388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.420233 5980 scope.go:122] "RemoveContainer" containerID="a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b" Oct 30 06:23:42.430256 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.430232 5980 scope.go:122] "RemoveContainer" containerID="9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc" Oct 30 06:23:42.439313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.439294 5980 scope.go:122] "RemoveContainer" containerID="a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b" Oct 30 06:23:42.439605 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:42.439586 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b\": container with ID starting with a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b not found: ID does not exist" containerID="a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b" Oct 30 06:23:42.439666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.439613 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b"} err="failed to get container status \"a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b\": rpc error: code = NotFound desc = could not find container \"a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b\": container with ID starting with a37f3d1ccd889a04f039371675a744ea823b904aa6ed79021641b6b5e88e967b not found: ID does not exist" Oct 30 06:23:42.439666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.439632 5980 scope.go:122] "RemoveContainer" containerID="9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc" Oct 30 06:23:42.439884 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:42.439861 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc\": container with ID starting with 9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc not found: ID does not exist" containerID="9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc" Oct 30 06:23:42.439938 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.439891 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc"} err="failed to get container status \"9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc\": rpc error: code = NotFound desc = could not find container \"9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc\": container with ID starting with 9131f24db2a0a6d777ea5533c4a87c88df2476ad8c63e6e97e8b2cd318236bdc not found: ID does not exist" Oct 30 06:23:42.490191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.490158 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp"] Oct 30 06:23:42.505390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:42.505354 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-s6vmp"] Oct 30 06:23:43.130130 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:43.116342 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:23:43.564618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.564526 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs"] Oct 30 06:23:43.565348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565322 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" Oct 30 06:23:43.565348 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565347 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565365 5980 cpu_manager.go:414] "containerMap: removing container" podUID="b41fd892-f798-4f67-8427-65f1601efa58" containerName="maas-api" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565373 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41fd892-f798-4f67-8427-65f1601efa58" containerName="maas-api" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565387 5980 cpu_manager.go:414] "containerMap: removing container" podUID="48abb063-1fc8-435b-af07-9fd4c43ed944" containerName="main" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565397 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48abb063-1fc8-435b-af07-9fd4c43ed944" containerName="main" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565415 5980 cpu_manager.go:414] "containerMap: removing container" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565422 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565433 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565440 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565453 5980 cpu_manager.go:414] "containerMap: removing container" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" containerName="payload-processing" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565460 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" containerName="payload-processing" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565484 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565491 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565510 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f1fc41cf-f675-4ada-b506-def7d09a8f41" containerName="maas-api" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565518 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fc41cf-f675-4ada-b506-def7d09a8f41" containerName="maas-api" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565532 5980 cpu_manager.go:414] "containerMap: removing container" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565539 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565552 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" Oct 30 06:23:43.565550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565560 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e709aeb7-5931-4b2d-a7a1-48f2f0e1adf8" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565571 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565580 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565590 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5fce3f95-3101-4a0b-9bed-1b4fafef341c" containerName="maas-api" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565598 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fce3f95-3101-4a0b-9bed-1b4fafef341c" containerName="maas-api" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565614 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565621 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565635 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a14d1d94-d361-4634-9cbc-cf4621f87cc1" containerName="payload-pre-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565643 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d1d94-d361-4634-9cbc-cf4621f87cc1" containerName="payload-pre-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565655 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565661 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565675 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerName="payload-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565681 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerName="payload-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565690 5980 cpu_manager.go:414] "containerMap: removing container" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565697 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377612ce-6b65-4347-949f-d03e31a4c596" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565706 5980 cpu_manager.go:414] "containerMap: removing container" podUID="39ed193f-7ac3-4428-9051-59a8e1b62b4e" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565713 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ed193f-7ac3-4428-9051-59a8e1b62b4e" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565725 5980 cpu_manager.go:414] "containerMap: removing container" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565732 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565745 5980 cpu_manager.go:414] "containerMap: removing container" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565752 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565765 5980 cpu_manager.go:414] "containerMap: removing container" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565771 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565783 5980 cpu_manager.go:414] "containerMap: removing container" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565790 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565889 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565901 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d1d94-d361-4634-9cbc-cf4621f87cc1" containerName="payload-pre-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565914 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38cd2fb-3ec8-4bb3-9fc4-8fd06f4eabd4" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565924 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda4eb6a-85f1-428e-8193-33da6f1482e4" containerName="payload-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565934 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ed193f-7ac3-4428-9051-59a8e1b62b4e" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565942 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4018a797-c102-4cbb-9e8d-8ff7a87098d6" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565951 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f49717-ed4b-4dd0-ad8f-8f1ffb40f21f" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565961 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb495806-9f5c-47d5-b035-f6b546928092" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565969 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565978 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41fd892-f798-4f67-8427-65f1601efa58" containerName="maas-api" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565987 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="48abb063-1fc8-435b-af07-9fd4c43ed944" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.565998 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a18b6a-88bf-43fd-b7a8-abae1840013d" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566006 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3ba20b-581d-4da1-8e15-ec98b13f08de" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566017 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d18805d-84cd-4b64-b910-0c0b14325230" containerName="payload-processing" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566027 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c48ddd4-40f4-41ed-9350-5e24c07ef671" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566038 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fc41cf-f675-4ada-b506-def7d09a8f41" containerName="maas-api" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566047 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cbe530-2176-4450-881b-74bd0a41eb1a" containerName="main" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566054 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a274fed5-ff91-44aa-a19f-129c6b9c8b32" containerName="istio-proxy" Oct 30 06:23:43.566537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.566062 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fce3f95-3101-4a0b-9bed-1b4fafef341c" containerName="maas-api" Oct 30 06:23:43.571390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.571366 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.598942 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.598917 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62"] Oct 30 06:23:43.599606 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.599585 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8723f03-8296-4c76-b138-6375e3561d45" containerName="istio-proxy" Oct 30 06:23:43.603446 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.603424 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.617307 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.617267 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-630fdf-openshift-default-dockercfg-gjmjl\"" Oct 30 06:23:43.619440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.619417 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs"] Oct 30 06:23:43.642283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.640443 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62"] Oct 30 06:23:43.658252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658227 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658383 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658260 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658383 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658376 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658487 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658408 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658487 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658431 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658487 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658450 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4zs\" (UniqueName: \"kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.658585 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658552 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.658624 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658591 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658620 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658750 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.658833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658777 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.658929 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.658906 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.719133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.718642 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:23:43.720311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.720158 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:43.760279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760001 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.760279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760053 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760125 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760180 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760279 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760208 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760306 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760349 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760387 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760408 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zs\" (UniqueName: \"kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760449 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760505 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.760736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.760540 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.761670 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.761645 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.772818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.762496 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.772818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.767589 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.772818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.767639 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.772818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.767883 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.772818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.771414 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.772818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.772681 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.775626 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.775581 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.778091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.778065 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.787651 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.787619 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.788684 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.788653 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld\") pod \"e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.798962 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.798927 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4zs\" (UniqueName: \"kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs\") pod \"maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:43.888314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.888283 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:43.930193 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:43.930160 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:44.020665 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.020613 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8723f03-8296-4c76-b138-6375e3561d45" path="/var/lib/kubelet/pods/f8723f03-8296-4c76-b138-6375e3561d45/volumes" Oct 30 06:23:44.218442 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:23:44.217491 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d23e03c_e12c_4013_920c_e368e861d998.slice/crio-5fcd390ea4a335c690cf5e462e9173cb48bd2fdc10265980b0ec910877365c7c WatchSource:0}: Error finding container 5fcd390ea4a335c690cf5e462e9173cb48bd2fdc10265980b0ec910877365c7c: Status 404 returned error can't find the container with id 5fcd390ea4a335c690cf5e462e9173cb48bd2fdc10265980b0ec910877365c7c Oct 30 06:23:44.218442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.218148 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs"] Oct 30 06:23:44.222717 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.222677 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:44.222818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.222765 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:44.222818 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.222810 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:44.269101 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.269070 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62"] Oct 30 06:23:44.277061 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:23:44.276939 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8c4dcf_e9dc_429e_88ca_b6f3a4c8ea67.slice/crio-6100a31157b057c3854c9874b74c84088f955a90b794be7ec80ded9eeb011db1 WatchSource:0}: Error finding container 6100a31157b057c3854c9874b74c84088f955a90b794be7ec80ded9eeb011db1: Status 404 returned error can't find the container with id 6100a31157b057c3854c9874b74c84088f955a90b794be7ec80ded9eeb011db1 Oct 30 06:23:44.311646 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.311612 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 06:23:44.376897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.376870 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.377129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377095 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.377244 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377235 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.377336 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377327 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.377467 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377454 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.377577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377568 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.378352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377657 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.378352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377724 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.378352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.377752 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs\") pod \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\" (UID: \"f96bb7f7-7077-4a32-9c20-0532cc7d6307\") " Oct 30 06:23:44.378642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.378618 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.379327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.379257 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.381056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.380858 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.381056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.380866 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.381056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.380934 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.381541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.381456 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:44.383447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.383300 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:44.383447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.383416 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "kube-api-access-rct6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:44.390134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.389741 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token" pod "f96bb7f7-7077-4a32-9c20-0532cc7d6307" (UID: "f96bb7f7-7077-4a32-9c20-0532cc7d6307"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:44.434202 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.434091 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" event={"ID":"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67","Type":"ContainerStarted","Data":"6100a31157b057c3854c9874b74c84088f955a90b794be7ec80ded9eeb011db1"} Oct 30 06:23:44.435938 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.435912 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" event={"ID":"7d23e03c-e12c-4013-920c-e368e861d998","Type":"ContainerStarted","Data":"23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941"} Oct 30 06:23:44.436087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.435955 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" event={"ID":"7d23e03c-e12c-4013-920c-e368e861d998","Type":"ContainerStarted","Data":"5fcd390ea4a335c690cf5e462e9173cb48bd2fdc10265980b0ec910877365c7c"} Oct 30 06:23:44.440756 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.440732 5980 generic.go:356] "Generic (PLEG): container finished" podID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerID="731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575" exitCode=0 Oct 30 06:23:44.440869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.440793 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" Oct 30 06:23:44.440917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.440794 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" event={"ID":"f96bb7f7-7077-4a32-9c20-0532cc7d6307","Type":"ContainerDied","Data":"731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575"} Oct 30 06:23:44.440952 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.440925 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz" event={"ID":"f96bb7f7-7077-4a32-9c20-0532cc7d6307","Type":"ContainerDied","Data":"4e0f1ce6803de652e33c23e4ee0a6f5e8b9348388d96963b740bfaa57a3da02c"} Oct 30 06:23:44.440996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.440954 5980 scope.go:122] "RemoveContainer" containerID="731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575" Oct 30 06:23:44.450893 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.450875 5980 scope.go:122] "RemoveContainer" containerID="731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575" Oct 30 06:23:44.451185 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:23:44.451167 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575\": container with ID starting with 731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575 not found: ID does not exist" containerID="731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575" Oct 30 06:23:44.451269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.451191 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575"} err="failed to get container status \"731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575\": rpc error: code = NotFound desc = could not find container \"731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575\": container with ID starting with 731a705880b34e18ea67afb3dc2c6fc61158c49241a995eac6e94d0d1e762575 not found: ID does not exist" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479799 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479830 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479843 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479856 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479870 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479884 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479897 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479909 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96bb7f7-7077-4a32-9c20-0532cc7d6307-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.480507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.479922 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rct6p\" (UniqueName: \"kubernetes.io/projected/f96bb7f7-7077-4a32-9c20-0532cc7d6307-kube-api-access-rct6p\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.483932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.483889 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podStartSLOduration=1.483877508 podStartE2EDuration="1.483877508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:44.482973594 +0000 UTC m=+3078.591012628" watchObservedRunningTime="2026-10-30 06:23:44.483877508 +0000 UTC m=+3078.591916540" Oct 30 06:23:44.522667 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.522638 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz"] Oct 30 06:23:44.537140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.535279 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-nx9gz"] Oct 30 06:23:44.889139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:44.889084 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:45.448847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:45.448810 5980 generic.go:356] "Generic (PLEG): container finished" podID="2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" containerID="f908b57b043bab5a872e211e71bcc17e9b45cab98b3f25d90bcd327cf7363882" exitCode=0 Oct 30 06:23:45.449350 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:45.448892 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" event={"ID":"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67","Type":"ContainerDied","Data":"f908b57b043bab5a872e211e71bcc17e9b45cab98b3f25d90bcd327cf7363882"} Oct 30 06:23:45.895297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:45.895264 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:46.015021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.014992 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" path="/var/lib/kubelet/pods/f96bb7f7-7077-4a32-9c20-0532cc7d6307/volumes" Oct 30 06:23:46.455481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.455452 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:46.456385 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.456362 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:23:46.640123 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.640087 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:46.709738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.709652 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs\" (UniqueName: \"kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs\") pod \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " Oct 30 06:23:46.709738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.709693 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca\") pod \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " Oct 30 06:23:46.710076 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.709752 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca\") pod \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\" (UID: \"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67\") " Oct 30 06:23:46.710327 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.710302 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca" pod "2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" (UID: "2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:46.712629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.712607 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca" pod "2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" (UID: "2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:46.712852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.712641 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs" pod "2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" (UID: "2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67"). InnerVolumeSpecName "kube-api-access-th4zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:46.811514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.811480 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th4zs\" (UniqueName: \"kubernetes.io/projected/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-kube-api-access-th4zs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.811514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.811510 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-xks-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.811514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:46.811520 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67-maas-api-openshift-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:23:47.461861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:47.461830 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" Oct 30 06:23:47.461861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:47.461839 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62" event={"ID":"2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67","Type":"ContainerDied","Data":"6100a31157b057c3854c9874b74c84088f955a90b794be7ec80ded9eeb011db1"} Oct 30 06:23:47.462433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:47.461890 5980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6100a31157b057c3854c9874b74c84088f955a90b794be7ec80ded9eeb011db1" Oct 30 06:23:48.718226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718191 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt"] Oct 30 06:23:48.718677 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718664 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" Oct 30 06:23:48.718724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718677 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" Oct 30 06:23:48.718724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718692 5980 cpu_manager.go:414] "containerMap: removing container" podUID="2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" containerName="revoke-keys" Oct 30 06:23:48.718724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718698 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" containerName="revoke-keys" Oct 30 06:23:48.718829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718773 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96bb7f7-7077-4a32-9c20-0532cc7d6307" containerName="istio-proxy" Oct 30 06:23:48.718829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.718783 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" containerName="revoke-keys" Oct 30 06:23:48.722328 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.722309 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:48.755763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.755739 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-630fdf\"" Oct 30 06:23:48.756100 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.756078 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-630fdf-dockercfg-bbcpc\"" Oct 30 06:23:48.759711 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.759687 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt"] Oct 30 06:23:48.776132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.776093 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5"] Oct 30 06:23:48.780140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.780006 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.802646 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.802610 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5"] Oct 30 06:23:48.872134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.869277 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66q4\" (UniqueName: \"kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4\") pod \"payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:48.872134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.869437 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtkq9\" (UniqueName: \"kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.872134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.869502 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:48.872134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.869533 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.970472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.970395 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:48.970472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.970438 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.970719 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.970584 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66q4\" (UniqueName: \"kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4\") pod \"payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:48.970719 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.970705 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtkq9\" (UniqueName: \"kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.971210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.971191 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:48.971325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.971307 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.995965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.995938 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtkq9\" (UniqueName: \"kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:48.996088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:48.995998 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66q4\" (UniqueName: \"kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4\") pod \"payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:49.036069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.036035 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:49.094511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.094479 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:49.342713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.342685 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt"] Oct 30 06:23:49.343698 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:23:49.343670 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c0923f_e276_4eb4_98fb_f1ebfec79a80.slice/crio-a7d1c258525e64329f23b2925bed146a38581829a9eece8e09a5c7d6328516c7 WatchSource:0}: Error finding container a7d1c258525e64329f23b2925bed146a38581829a9eece8e09a5c7d6328516c7: Status 404 returned error can't find the container with id a7d1c258525e64329f23b2925bed146a38581829a9eece8e09a5c7d6328516c7 Oct 30 06:23:49.423169 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:23:49.423143 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842f5972_60c1_4b1b_ab62_cf15f2c0ee05.slice/crio-63f294b6fb09e9152530e5e330003d6c7ea352bf224caea69248942e9d6ad5db WatchSource:0}: Error finding container 63f294b6fb09e9152530e5e330003d6c7ea352bf224caea69248942e9d6ad5db: Status 404 returned error can't find the container with id 63f294b6fb09e9152530e5e330003d6c7ea352bf224caea69248942e9d6ad5db Oct 30 06:23:49.424529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.424459 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5"] Oct 30 06:23:49.475341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.475312 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" event={"ID":"842f5972-60c1-4b1b-ab62-cf15f2c0ee05","Type":"ContainerStarted","Data":"63f294b6fb09e9152530e5e330003d6c7ea352bf224caea69248942e9d6ad5db"} Oct 30 06:23:49.477054 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.477027 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" event={"ID":"42c0923f-e276-4eb4-98fb-f1ebfec79a80","Type":"ContainerStarted","Data":"22a11b83dafb5d33cb0ae460d25aa176cbce565eb65229f561a973227e4f42f1"} Oct 30 06:23:49.477197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.477078 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" event={"ID":"42c0923f-e276-4eb4-98fb-f1ebfec79a80","Type":"ContainerStarted","Data":"a7d1c258525e64329f23b2925bed146a38581829a9eece8e09a5c7d6328516c7"} Oct 30 06:23:49.477197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.477153 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:23:49.523322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:49.523271 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" podStartSLOduration=1.5232570079999999 podStartE2EDuration="1.523257008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:49.521526016 +0000 UTC m=+3083.629565048" watchObservedRunningTime="2026-10-30 06:23:49.523257008 +0000 UTC m=+3083.631296041" Oct 30 06:23:50.484762 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:50.484724 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" event={"ID":"842f5972-60c1-4b1b-ab62-cf15f2c0ee05","Type":"ContainerStarted","Data":"2ff1692cd2d1f3ed76dbd059b8a6af300358f0fadd5d269572b44dbbd3180963"} Oct 30 06:23:50.485829 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:50.484894 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:23:50.528481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:50.528430 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" podStartSLOduration=2.528417401 podStartE2EDuration="2.528417401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:50.528260983 +0000 UTC m=+3084.636299998" watchObservedRunningTime="2026-10-30 06:23:50.528417401 +0000 UTC m=+3084.636456440" Oct 30 06:23:56.054560 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:56.054529 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62"] Oct 30 06:23:56.070570 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:56.070535 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d51cd8-gxf62"] Oct 30 06:23:58.012580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:23:58.012546 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67" path="/var/lib/kubelet/pods/2f8c4dcf-e9dc-429e-88ca-b6f3a4c8ea67/volumes" Oct 30 06:24:00.488443 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:00.488417 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:24:01.492993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:01.492962 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:24:03.421923 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.421888 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm"] Oct 30 06:24:03.422367 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.422167 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" containerID="cri-o://1887a0cc290a062bd2077fabae32e7208bbdbd581c8d94eb042d794746a896fc" gracePeriod=30 Oct 30 06:24:03.435209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.435179 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf"] Oct 30 06:24:03.435442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.435410 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" containerID="cri-o://9dad03d92625eac23229c29f71b18ea8f8807aaec2fca72eb79aabb6e6669d00" gracePeriod=30 Oct 30 06:24:03.437266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.437226 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9"] Oct 30 06:24:03.437433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.437408 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" containerID="cri-o://4b37efee3062c2cc3d9d45f873ca29545df4e001fbb681975ca67a462e4d43aa" gracePeriod=30 Oct 30 06:24:03.453574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.453549 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf"] Oct 30 06:24:03.575186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:03.575148 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" containerID="cri-o://f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e" gracePeriod=30 Oct 30 06:24:08.256686 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.256654 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm"] Oct 30 06:24:08.257298 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.256920 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" containerID="cri-o://2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9" gracePeriod=30 Oct 30 06:24:08.522967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.522885 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:08.522967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.522949 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:08.613518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.613307 5980 generic.go:356] "Generic (PLEG): container finished" podID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerID="1887a0cc290a062bd2077fabae32e7208bbdbd581c8d94eb042d794746a896fc" exitCode=0 Oct 30 06:24:08.613518 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.613427 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" event={"ID":"fc7c8255-55fa-4331-b9cc-d66298842b37","Type":"ContainerDied","Data":"1887a0cc290a062bd2077fabae32e7208bbdbd581c8d94eb042d794746a896fc"} Oct 30 06:24:08.629427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.622945 5980 generic.go:356] "Generic (PLEG): container finished" podID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerID="4b37efee3062c2cc3d9d45f873ca29545df4e001fbb681975ca67a462e4d43aa" exitCode=0 Oct 30 06:24:08.629427 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.623260 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" event={"ID":"26f52cb4-299f-465f-a8aa-c980be86bbfc","Type":"ContainerDied","Data":"4b37efee3062c2cc3d9d45f873ca29545df4e001fbb681975ca67a462e4d43aa"} Oct 30 06:24:08.636147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.632805 5980 generic.go:356] "Generic (PLEG): container finished" podID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerID="9dad03d92625eac23229c29f71b18ea8f8807aaec2fca72eb79aabb6e6669d00" exitCode=0 Oct 30 06:24:08.636147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.632858 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" event={"ID":"d231cd83-486c-4e4c-a623-4519bbfa2a37","Type":"ContainerDied","Data":"9dad03d92625eac23229c29f71b18ea8f8807aaec2fca72eb79aabb6e6669d00"} Oct 30 06:24:08.735702 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.735557 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:24:08.801628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.800555 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:24:08.821517 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.821492 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:24:08.852035 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.851991 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852071 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852096 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852138 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852179 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852237 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852223 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852241 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852265 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852287 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852319 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852335 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852367 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852393 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852428 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852456 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852475 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852492 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852518 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852548 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852566 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852600 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852634 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852651 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852685 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.852724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852715 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data\") pod \"fc7c8255-55fa-4331-b9cc-d66298842b37\" (UID: \"fc7c8255-55fa-4331-b9cc-d66298842b37\") " Oct 30 06:24:08.853292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852749 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs\") pod \"d231cd83-486c-4e4c-a623-4519bbfa2a37\" (UID: \"d231cd83-486c-4e4c-a623-4519bbfa2a37\") " Oct 30 06:24:08.853292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.852809 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs\") pod \"26f52cb4-299f-465f-a8aa-c980be86bbfc\" (UID: \"26f52cb4-299f-465f-a8aa-c980be86bbfc\") " Oct 30 06:24:08.853619 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.853590 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.854422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.854293 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.855698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.855410 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.856091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.855857 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.856470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.856267 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.856609 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.858293 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.858831 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.859011 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.859215 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.859902 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.860609 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.861015 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.861618 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.864872 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.862871 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.865798 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.865747 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.866015 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.865987 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.867930 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.867430 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.867930 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.867456 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.867930 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.867627 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "kube-api-access-rj57r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.867930 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.867816 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.867930 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.867904 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "kube-api-access-pcvnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.869281 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.869260 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.869778 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.869745 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token" pod "d231cd83-486c-4e4c-a623-4519bbfa2a37" (UID: "d231cd83-486c-4e4c-a623-4519bbfa2a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.869951 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.869932 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo" pod "26f52cb4-299f-465f-a8aa-c980be86bbfc" (UID: "26f52cb4-299f-465f-a8aa-c980be86bbfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.870888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.870868 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.872457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.872423 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd" pod "fc7c8255-55fa-4331-b9cc-d66298842b37" (UID: "fc7c8255-55fa-4331-b9cc-d66298842b37"). InnerVolumeSpecName "kube-api-access-k47rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.895191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.895001 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:24:08.955392 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955335 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955417 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955533 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955592 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955621 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955653 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955713 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955754 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.955943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.955805 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data\") pod \"47573111-9bc0-43e8-b490-f6d908e928fe\" (UID: \"47573111-9bc0-43e8-b490-f6d908e928fe\") " Oct 30 06:24:08.956304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956284 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956310 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f52cb4-299f-465f-a8aa-c980be86bbfc-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956323 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956336 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956352 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956364 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956377 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956389 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47rd\" (UniqueName: \"kubernetes.io/projected/fc7c8255-55fa-4331-b9cc-d66298842b37-kube-api-access-k47rd\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956401 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956415 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956428 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956441 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj57r\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-kube-api-access-rj57r\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956453 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956465 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.956480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956477 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d231cd83-486c-4e4c-a623-4519bbfa2a37-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956488 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956501 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956516 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956529 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956541 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc7c8255-55fa-4331-b9cc-d66298842b37-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956554 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvnn\" (UniqueName: \"kubernetes.io/projected/26f52cb4-299f-465f-a8aa-c980be86bbfc-kube-api-access-pcvnn\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956569 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d231cd83-486c-4e4c-a623-4519bbfa2a37-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956583 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc7c8255-55fa-4331-b9cc-d66298842b37-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956598 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f52cb4-299f-465f-a8aa-c980be86bbfc-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956611 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956626 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d231cd83-486c-4e4c-a623-4519bbfa2a37-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.959371 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.956639 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc7c8255-55fa-4331-b9cc-d66298842b37-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.961365 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.961001 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.961365 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.961326 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.963365 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.963326 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.967207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.966861 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.967207 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.967042 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.970499 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.970462 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.971577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.971551 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.971698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.971632 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.973602 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:08.973566 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8" pod "47573111-9bc0-43e8-b490-f6d908e928fe" (UID: "47573111-9bc0-43e8-b490-f6d908e928fe"). InnerVolumeSpecName "kube-api-access-t8pq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057802 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057831 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057841 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8pq8\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-kube-api-access-t8pq8\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057851 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47573111-9bc0-43e8-b490-f6d908e928fe-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057859 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47573111-9bc0-43e8-b490-f6d908e928fe-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057866 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47573111-9bc0-43e8-b490-f6d908e928fe-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.057870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057875 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.058235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057883 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.058235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.057895 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47573111-9bc0-43e8-b490-f6d908e928fe-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.641871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.641832 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" event={"ID":"fc7c8255-55fa-4331-b9cc-d66298842b37","Type":"ContainerDied","Data":"e668bcc7ac5069ce0388dbeddf70f877abc6f012f01100a9312b0f73e0969762"} Oct 30 06:24:09.641871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.641861 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm" Oct 30 06:24:09.642486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.641922 5980 scope.go:122] "RemoveContainer" containerID="1887a0cc290a062bd2077fabae32e7208bbdbd581c8d94eb042d794746a896fc" Oct 30 06:24:09.643470 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.643406 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" Oct 30 06:24:09.643561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.643478 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9" event={"ID":"26f52cb4-299f-465f-a8aa-c980be86bbfc","Type":"ContainerDied","Data":"e46792f48091950f97c5c6039009da1eac6188209e9b7cb02598448d20059428"} Oct 30 06:24:09.649757 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.649734 5980 generic.go:356] "Generic (PLEG): container finished" podID="47573111-9bc0-43e8-b490-f6d908e928fe" containerID="f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e" exitCode=0 Oct 30 06:24:09.649901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.649797 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" Oct 30 06:24:09.649901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.649823 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" event={"ID":"47573111-9bc0-43e8-b490-f6d908e928fe","Type":"ContainerDied","Data":"f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e"} Oct 30 06:24:09.649901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.649869 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf" event={"ID":"47573111-9bc0-43e8-b490-f6d908e928fe","Type":"ContainerDied","Data":"322bfebf979ced9a5968af1ad1e9f47d7272d5416e1ae14f1a1ee89817f9d62b"} Oct 30 06:24:09.651540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.651503 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" event={"ID":"d231cd83-486c-4e4c-a623-4519bbfa2a37","Type":"ContainerDied","Data":"c7b6673c3069912d12ae4aa2ab0d7864088a88479a094ae2d2b3a8785201f252"} Oct 30 06:24:09.651658 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.651514 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf" Oct 30 06:24:09.654502 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.654487 5980 scope.go:122] "RemoveContainer" containerID="4b37efee3062c2cc3d9d45f873ca29545df4e001fbb681975ca67a462e4d43aa" Oct 30 06:24:09.666615 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.666596 5980 scope.go:122] "RemoveContainer" containerID="f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e" Oct 30 06:24:09.676540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.676517 5980 scope.go:122] "RemoveContainer" containerID="ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0" Oct 30 06:24:09.687736 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.687707 5980 scope.go:122] "RemoveContainer" containerID="f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e" Oct 30 06:24:09.688060 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:09.688035 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e\": container with ID starting with f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e not found: ID does not exist" containerID="f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e" Oct 30 06:24:09.688236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.688070 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e"} err="failed to get container status \"f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e\": rpc error: code = NotFound desc = could not find container \"f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e\": container with ID starting with f548d7f7abbeda8b7c68bc535c992defd5298def6f7c892dd5b850bf5f81aa8e not found: ID does not exist" Oct 30 06:24:09.688236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.688099 5980 scope.go:122] "RemoveContainer" containerID="ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0" Oct 30 06:24:09.689059 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:09.688398 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0\": container with ID starting with ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0 not found: ID does not exist" containerID="ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0" Oct 30 06:24:09.689059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.688422 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0"} err="failed to get container status \"ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0\": rpc error: code = NotFound desc = could not find container \"ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0\": container with ID starting with ed8cbb8eb67080b5fcf3367ab4541692fc8a804c5801e0e27b0b3cf58fe5cbe0 not found: ID does not exist" Oct 30 06:24:09.689059 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.688436 5980 scope.go:122] "RemoveContainer" containerID="9dad03d92625eac23229c29f71b18ea8f8807aaec2fca72eb79aabb6e6669d00" Oct 30 06:24:09.719897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.719800 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9"] Oct 30 06:24:09.727160 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.727131 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-4s4d9"] Oct 30 06:24:09.756916 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.756883 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf"] Oct 30 06:24:09.760997 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.760968 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-g28wf"] Oct 30 06:24:09.802593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.802566 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf"] Oct 30 06:24:09.806688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.806661 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-x9fsf"] Oct 30 06:24:09.843223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.843185 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm"] Oct 30 06:24:09.847507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:09.847482 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-j2msm"] Oct 30 06:24:10.013663 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:10.013592 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" path="/var/lib/kubelet/pods/26f52cb4-299f-465f-a8aa-c980be86bbfc/volumes" Oct 30 06:24:10.014028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:10.014015 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" path="/var/lib/kubelet/pods/47573111-9bc0-43e8-b490-f6d908e928fe/volumes" Oct 30 06:24:10.014459 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:10.014446 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" path="/var/lib/kubelet/pods/d231cd83-486c-4e4c-a623-4519bbfa2a37/volumes" Oct 30 06:24:10.015194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:10.015179 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" path="/var/lib/kubelet/pods/fc7c8255-55fa-4331-b9cc-d66298842b37/volumes" Oct 30 06:24:13.087316 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:13.087284 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:24:13.535622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.535598 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 06:24:13.604726 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.604693 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.604744 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.604793 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.604850 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.604882 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.604962 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605005 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605057 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605129 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605193 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket\") pod \"e2309786-9bb4-4274-962f-47f3697e5309\" (UID: \"e2309786-9bb4-4274-962f-47f3697e5309\") " Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605681 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605717 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:13.605968 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605785 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:13.606718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.605982 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:13.606718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.606047 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:13.608301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.608272 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:13.609640 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.609619 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "kube-api-access-66khp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:13.609766 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.609642 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:13.622152 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.622122 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy" pod "e2309786-9bb4-4274-962f-47f3697e5309" (UID: "e2309786-9bb4-4274-962f-47f3697e5309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:13.680974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.680940 5980 generic.go:356] "Generic (PLEG): container finished" podID="e2309786-9bb4-4274-962f-47f3697e5309" containerID="2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9" exitCode=0 Oct 30 06:24:13.681186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.680994 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" Oct 30 06:24:13.681186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.681022 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" event={"ID":"e2309786-9bb4-4274-962f-47f3697e5309","Type":"ContainerDied","Data":"2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9"} Oct 30 06:24:13.681186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.681091 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm" event={"ID":"e2309786-9bb4-4274-962f-47f3697e5309","Type":"ContainerDied","Data":"7e2b5c8287fc0e27be0a4bfb03e47e9494fc91664a2a894bf2b1f5c03fa020d0"} Oct 30 06:24:13.681186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.681143 5980 scope.go:122] "RemoveContainer" containerID="2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9" Oct 30 06:24:13.693116 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.693086 5980 scope.go:122] "RemoveContainer" containerID="24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94" Oct 30 06:24:13.702975 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.702957 5980 scope.go:122] "RemoveContainer" containerID="2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9" Oct 30 06:24:13.703281 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:13.703259 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9\": container with ID starting with 2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9 not found: ID does not exist" containerID="2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9" Oct 30 06:24:13.703366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.703290 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9"} err="failed to get container status \"2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9\": rpc error: code = NotFound desc = could not find container \"2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9\": container with ID starting with 2f4f39eed25a68d5bc6d9a60c368b9fe148085a0ca79147cef96d00b180861b9 not found: ID does not exist" Oct 30 06:24:13.703366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.703308 5980 scope.go:122] "RemoveContainer" containerID="24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94" Oct 30 06:24:13.703595 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:13.703561 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94\": container with ID starting with 24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94 not found: ID does not exist" containerID="24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94" Oct 30 06:24:13.703634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.703606 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94"} err="failed to get container status \"24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94\": rpc error: code = NotFound desc = could not find container \"24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94\": container with ID starting with 24722bcb87eb1780e4d7122238cae3640bbe9cfe97c5a97aaf68bea44de3df94 not found: ID does not exist" Oct 30 06:24:13.707195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707171 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66khp\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-kube-api-access-66khp\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707197 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2309786-9bb4-4274-962f-47f3697e5309-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707211 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707223 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707236 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2309786-9bb4-4274-962f-47f3697e5309-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707248 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707260 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2309786-9bb4-4274-962f-47f3697e5309-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.707292 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.707274 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2309786-9bb4-4274-962f-47f3697e5309-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:13.737592 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.737571 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm"] Oct 30 06:24:13.742155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:13.742132 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-lj8hm"] Oct 30 06:24:14.014631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:14.014546 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2309786-9bb4-4274-962f-47f3697e5309" path="/var/lib/kubelet/pods/e2309786-9bb4-4274-962f-47f3697e5309/volumes" Oct 30 06:24:26.763189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:26.763075 5980 generic.go:356] "Generic (PLEG): container finished" podID="7d23e03c-e12c-4013-920c-e368e861d998" containerID="23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941" exitCode=0 Oct 30 06:24:26.763189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:26.763156 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" event={"ID":"7d23e03c-e12c-4013-920c-e368e861d998","Type":"ContainerDied","Data":"23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941"} Oct 30 06:24:26.763631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:26.763559 5980 scope.go:122] "RemoveContainer" containerID="23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941" Oct 30 06:24:26.765056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:26.765026 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:26.765186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:26.765098 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:26.765186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:26.765152 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:27.484880 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:27.484246 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt"] Oct 30 06:24:27.517065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:27.517024 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5"] Oct 30 06:24:27.795167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:27.795035 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" event={"ID":"7d23e03c-e12c-4013-920c-e368e861d998","Type":"ContainerStarted","Data":"020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb"} Oct 30 06:24:27.890285 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:27.890180 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:24:27.890967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:27.890638 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:24:27.890967 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:27.890688 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 06:24:28.889665 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:28.889633 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:24:28.890198 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:28.889688 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 06:24:29.107985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.093627 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w"] Oct 30 06:24:29.107985 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.093875 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" podUID="d8282f0e-56e5-45ca-afa6-db63c4fea78a" containerName="maas-api" containerID="cri-o://a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6" gracePeriod=30 Oct 30 06:24:29.450657 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.450319 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 06:24:29.625794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.624904 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\") pod \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " Oct 30 06:24:29.625794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.625023 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca\") pod \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " Oct 30 06:24:29.625794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.625075 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs\") pod \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " Oct 30 06:24:29.625794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.625178 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2\" (UniqueName: \"kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2\") pod \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\" (UID: \"d8282f0e-56e5-45ca-afa6-db63c4fea78a\") " Oct 30 06:24:29.625794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.625433 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca" pod "d8282f0e-56e5-45ca-afa6-db63c4fea78a" (UID: "d8282f0e-56e5-45ca-afa6-db63c4fea78a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:29.625794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.625627 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8282f0e-56e5-45ca-afa6-db63c4fea78a-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.628804 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.628772 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs" pod "d8282f0e-56e5-45ca-afa6-db63c4fea78a" (UID: "d8282f0e-56e5-45ca-afa6-db63c4fea78a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:29.629614 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.629552 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2" pod "d8282f0e-56e5-45ca-afa6-db63c4fea78a" (UID: "d8282f0e-56e5-45ca-afa6-db63c4fea78a"). InnerVolumeSpecName "kube-api-access-t5zl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:29.631302 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.631280 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls" pod "d8282f0e-56e5-45ca-afa6-db63c4fea78a" (UID: "d8282f0e-56e5-45ca-afa6-db63c4fea78a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:29.726378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.726283 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zl2\" (UniqueName: \"kubernetes.io/projected/d8282f0e-56e5-45ca-afa6-db63c4fea78a-kube-api-access-t5zl2\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.726378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.726317 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.726378 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.726327 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8282f0e-56e5-45ca-afa6-db63c4fea78a-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:29.814131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.809215 5980 generic.go:356] "Generic (PLEG): container finished" podID="d8282f0e-56e5-45ca-afa6-db63c4fea78a" containerID="a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6" exitCode=0 Oct 30 06:24:29.814131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.809282 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" Oct 30 06:24:29.814131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.809363 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" event={"ID":"d8282f0e-56e5-45ca-afa6-db63c4fea78a","Type":"ContainerDied","Data":"a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6"} Oct 30 06:24:29.814131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.809399 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w" event={"ID":"d8282f0e-56e5-45ca-afa6-db63c4fea78a","Type":"ContainerDied","Data":"3ca3d4a87d4656afcfdafa83214c65a466c1c8ca1d017ae367fdef6161a7996b"} Oct 30 06:24:29.814131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.809426 5980 scope.go:122] "RemoveContainer" containerID="a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6" Oct 30 06:24:29.833351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.831925 5980 scope.go:122] "RemoveContainer" containerID="a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6" Oct 30 06:24:29.833351 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:29.832412 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6\": container with ID starting with a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6 not found: ID does not exist" containerID="a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6" Oct 30 06:24:29.833351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.832451 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6"} err="failed to get container status \"a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6\": rpc error: code = NotFound desc = could not find container \"a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6\": container with ID starting with a5078fe7c03b28695d4c30eb0314c14d977d0dedf3a21becd4dceca3fb314ee6 not found: ID does not exist" Oct 30 06:24:29.884134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.883676 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w"] Oct 30 06:24:29.893132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.889471 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:24:29.893132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.889521 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 06:24:29.894772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:29.894038 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-815776-54cf7c64bc-kdt9w"] Oct 30 06:24:30.012140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:30.012026 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8282f0e-56e5-45ca-afa6-db63c4fea78a" path="/var/lib/kubelet/pods/d8282f0e-56e5-45ca-afa6-db63c4fea78a/volumes" Oct 30 06:24:30.890131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:30.889236 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:24:30.890131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:30.889299 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 06:24:31.456157 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:31.456096 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:24:31.889671 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:31.889638 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:24:31.889850 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:31.889689 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 06:24:32.489387 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.489176 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" podUID="42c0923f-e276-4eb4-98fb-f1ebfec79a80" containerName="payload-processing" containerID="cri-o://22a11b83dafb5d33cb0ae460d25aa176cbce565eb65229f561a973227e4f42f1" gracePeriod=25 Oct 30 06:24:32.518221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.518167 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" podUID="842f5972-60c1-4b1b-ab62-cf15f2c0ee05" containerName="payload-pre-processing" containerID="cri-o://2ff1692cd2d1f3ed76dbd059b8a6af300358f0fadd5d269572b44dbbd3180963" gracePeriod=25 Oct 30 06:24:32.838359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.838231 5980 generic.go:356] "Generic (PLEG): container finished" podID="842f5972-60c1-4b1b-ab62-cf15f2c0ee05" containerID="2ff1692cd2d1f3ed76dbd059b8a6af300358f0fadd5d269572b44dbbd3180963" exitCode=0 Oct 30 06:24:32.838359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.838347 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" event={"ID":"842f5972-60c1-4b1b-ab62-cf15f2c0ee05","Type":"ContainerDied","Data":"2ff1692cd2d1f3ed76dbd059b8a6af300358f0fadd5d269572b44dbbd3180963"} Oct 30 06:24:32.840229 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.840148 5980 generic.go:356] "Generic (PLEG): container finished" podID="42c0923f-e276-4eb4-98fb-f1ebfec79a80" containerID="22a11b83dafb5d33cb0ae460d25aa176cbce565eb65229f561a973227e4f42f1" exitCode=0 Oct 30 06:24:32.840317 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.840233 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" event={"ID":"42c0923f-e276-4eb4-98fb-f1ebfec79a80","Type":"ContainerDied","Data":"22a11b83dafb5d33cb0ae460d25aa176cbce565eb65229f561a973227e4f42f1"} Oct 30 06:24:32.865574 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.865272 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:24:32.889278 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.889242 5980 patch_prober.go:29] interesting pod/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 06:24:32.889562 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.889534 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 06:24:32.921718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.921691 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:24:32.954090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.954061 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume\") pod \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " Oct 30 06:24:32.954246 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.954123 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume\") pod \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " Oct 30 06:24:32.954246 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.954178 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4\" (UniqueName: \"kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4\") pod \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\" (UID: \"42c0923f-e276-4eb4-98fb-f1ebfec79a80\") " Oct 30 06:24:32.954246 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.954219 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9\" (UniqueName: \"kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9\") pod \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\" (UID: \"842f5972-60c1-4b1b-ab62-cf15f2c0ee05\") " Oct 30 06:24:32.954556 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.954527 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume" pod "842f5972-60c1-4b1b-ab62-cf15f2c0ee05" (UID: "842f5972-60c1-4b1b-ab62-cf15f2c0ee05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:32.954628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.954563 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume" pod "42c0923f-e276-4eb4-98fb-f1ebfec79a80" (UID: "42c0923f-e276-4eb4-98fb-f1ebfec79a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:32.956537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.956513 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9" pod "842f5972-60c1-4b1b-ab62-cf15f2c0ee05" (UID: "842f5972-60c1-4b1b-ab62-cf15f2c0ee05"). InnerVolumeSpecName "kube-api-access-rtkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:32.956627 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:32.956543 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4" pod "42c0923f-e276-4eb4-98fb-f1ebfec79a80" (UID: "42c0923f-e276-4eb4-98fb-f1ebfec79a80"). InnerVolumeSpecName "kube-api-access-c66q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:33.055713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.055674 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:33.055713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.055710 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c0923f-e276-4eb4-98fb-f1ebfec79a80-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:33.055713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.055722 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66q4\" (UniqueName: \"kubernetes.io/projected/42c0923f-e276-4eb4-98fb-f1ebfec79a80-kube-api-access-c66q4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:33.056002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.055731 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtkq9\" (UniqueName: \"kubernetes.io/projected/842f5972-60c1-4b1b-ab62-cf15f2c0ee05-kube-api-access-rtkq9\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:33.846970 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.846943 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" Oct 30 06:24:33.847462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.846972 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5" event={"ID":"842f5972-60c1-4b1b-ab62-cf15f2c0ee05","Type":"ContainerDied","Data":"63f294b6fb09e9152530e5e330003d6c7ea352bf224caea69248942e9d6ad5db"} Oct 30 06:24:33.847462 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.847046 5980 scope.go:122] "RemoveContainer" containerID="2ff1692cd2d1f3ed76dbd059b8a6af300358f0fadd5d269572b44dbbd3180963" Oct 30 06:24:33.848333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.848291 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" event={"ID":"42c0923f-e276-4eb4-98fb-f1ebfec79a80","Type":"ContainerDied","Data":"a7d1c258525e64329f23b2925bed146a38581829a9eece8e09a5c7d6328516c7"} Oct 30 06:24:33.848394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.848319 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt" Oct 30 06:24:33.856995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.856977 5980 scope.go:122] "RemoveContainer" containerID="22a11b83dafb5d33cb0ae460d25aa176cbce565eb65229f561a973227e4f42f1" Oct 30 06:24:33.893366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.893343 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:24:33.894372 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.894343 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:24:33.902541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.902512 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt"] Oct 30 06:24:33.916186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.916166 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-5fdf864f85-gd7lt"] Oct 30 06:24:33.944226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.944206 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5"] Oct 30 06:24:33.949385 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:33.949365 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-54fc8f757f-6bgd5"] Oct 30 06:24:34.022151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:34.016433 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c0923f-e276-4eb4-98fb-f1ebfec79a80" path="/var/lib/kubelet/pods/42c0923f-e276-4eb4-98fb-f1ebfec79a80/volumes" Oct 30 06:24:34.022151 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:34.017012 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842f5972-60c1-4b1b-ab62-cf15f2c0ee05" path="/var/lib/kubelet/pods/842f5972-60c1-4b1b-ab62-cf15f2c0ee05/volumes" Oct 30 06:24:37.338210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:37.338177 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r"] Oct 30 06:24:37.338698 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:37.338427 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" containerID="cri-o://6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325" gracePeriod=30 Oct 30 06:24:37.403913 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:37.403878 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:37.404073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:37.403926 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:39.603754 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.603710 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2"] Oct 30 06:24:39.605262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605236 5980 cpu_manager.go:414] "containerMap: removing container" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" Oct 30 06:24:39.605262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605261 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605284 5980 cpu_manager.go:414] "containerMap: removing container" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605292 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605312 5980 cpu_manager.go:414] "containerMap: removing container" podUID="842f5972-60c1-4b1b-ab62-cf15f2c0ee05" containerName="payload-pre-processing" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605320 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842f5972-60c1-4b1b-ab62-cf15f2c0ee05" containerName="payload-pre-processing" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605336 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d8282f0e-56e5-45ca-afa6-db63c4fea78a" containerName="maas-api" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605343 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8282f0e-56e5-45ca-afa6-db63c4fea78a" containerName="maas-api" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605362 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605369 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605382 5980 cpu_manager.go:414] "containerMap: removing container" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605389 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605401 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605408 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605423 5980 cpu_manager.go:414] "containerMap: removing container" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605430 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605440 5980 cpu_manager.go:414] "containerMap: removing container" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605447 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605466 5980 cpu_manager.go:414] "containerMap: removing container" podUID="42c0923f-e276-4eb4-98fb-f1ebfec79a80" containerName="payload-processing" Oct 30 06:24:39.605479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605473 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c0923f-e276-4eb4-98fb-f1ebfec79a80" containerName="payload-processing" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605565 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605576 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="842f5972-60c1-4b1b-ab62-cf15f2c0ee05" containerName="payload-pre-processing" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605586 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8282f0e-56e5-45ca-afa6-db63c4fea78a" containerName="maas-api" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605599 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605608 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc7c8255-55fa-4331-b9cc-d66298842b37" containerName="istio-proxy" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605617 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2309786-9bb4-4274-962f-47f3697e5309" containerName="istio-proxy" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605626 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f52cb4-299f-465f-a8aa-c980be86bbfc" containerName="istio-proxy" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605635 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d231cd83-486c-4e4c-a623-4519bbfa2a37" containerName="istio-proxy" Oct 30 06:24:39.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.605644 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c0923f-e276-4eb4-98fb-f1ebfec79a80" containerName="payload-processing" Oct 30 06:24:39.610785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.610763 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.627536 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.627512 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-630fdf-dockercfg-t9lbz\"" Oct 30 06:24:39.630011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.629988 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2"] Oct 30 06:24:39.640861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.640829 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-630fdf\"" Oct 30 06:24:39.728526 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.728496 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8m6d\" (UniqueName: \"kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.728675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.728542 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.765690 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.765661 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6"] Oct 30 06:24:39.766239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.766220 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="47573111-9bc0-43e8-b490-f6d908e928fe" containerName="istio-proxy" Oct 30 06:24:39.769621 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.769605 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:39.797170 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.797145 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6"] Oct 30 06:24:39.829368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.829344 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:39.829520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.829392 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m6d\" (UniqueName: \"kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.829520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.829488 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.829639 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.829542 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w46jz\" (UniqueName: \"kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz\") pod \"payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:39.830195 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.830175 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.854697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.854645 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8m6d\" (UniqueName: \"kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d\") pod \"payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.925581 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.925554 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:39.931264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.931233 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w46jz\" (UniqueName: \"kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz\") pod \"payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:39.931416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.931395 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:39.932032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.932015 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:39.956196 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:39.956172 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w46jz\" (UniqueName: \"kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz\") pod \"payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:40.084090 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:40.084060 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:40.233659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:40.233628 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2"] Oct 30 06:24:40.234816 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:24:40.234789 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed5e800_4476_4101_814f_27fa5caaba65.slice/crio-67699499ae22424f275a157fc955e79bf785dd1f0e6d23d24c8f3611f3b105ed WatchSource:0}: Error finding container 67699499ae22424f275a157fc955e79bf785dd1f0e6d23d24c8f3611f3b105ed: Status 404 returned error can't find the container with id 67699499ae22424f275a157fc955e79bf785dd1f0e6d23d24c8f3611f3b105ed Oct 30 06:24:40.393638 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:40.393608 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6"] Oct 30 06:24:40.394283 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:24:40.394245 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187669b7_aa6f_4124_8351_2dd33b9a19d6.slice/crio-c43cb192e53cedbebb43583e77e211bbb9cf9c531ce55a8fef96e4b0bb9dff50 WatchSource:0}: Error finding container c43cb192e53cedbebb43583e77e211bbb9cf9c531ce55a8fef96e4b0bb9dff50: Status 404 returned error can't find the container with id c43cb192e53cedbebb43583e77e211bbb9cf9c531ce55a8fef96e4b0bb9dff50 Oct 30 06:24:40.910914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:40.910813 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" event={"ID":"187669b7-aa6f-4124-8351-2dd33b9a19d6","Type":"ContainerStarted","Data":"c43cb192e53cedbebb43583e77e211bbb9cf9c531ce55a8fef96e4b0bb9dff50"} Oct 30 06:24:40.913263 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:40.913232 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" event={"ID":"bed5e800-4476-4101-814f-27fa5caaba65","Type":"ContainerStarted","Data":"67699499ae22424f275a157fc955e79bf785dd1f0e6d23d24c8f3611f3b105ed"} Oct 30 06:24:42.626004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.625978 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 06:24:42.661847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.661817 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.661984 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.661911 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.661984 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.661965 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662015 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662069 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662058 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662086 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.662209 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662152 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662230 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662289 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662324 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662301 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.662503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662326 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk\") pod \"7011a201-0f2a-490c-80fe-e9b5cea5398e\" (UID: \"7011a201-0f2a-490c-80fe-e9b5cea5398e\") " Oct 30 06:24:42.662564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662520 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.662631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662554 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.662779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662702 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.662779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662728 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.662779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662744 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.662779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662757 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.663037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.662883 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:42.664848 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.664813 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:42.665178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.665156 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "kube-api-access-ldxhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:42.665402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.665381 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.666087 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.666068 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo" pod "7011a201-0f2a-490c-80fe-e9b5cea5398e" (UID: "7011a201-0f2a-490c-80fe-e9b5cea5398e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:42.763869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.763838 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.763869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.763864 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7011a201-0f2a-490c-80fe-e9b5cea5398e-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.763869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.763874 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.764175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.763882 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.764175 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.763890 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldxhk\" (UniqueName: \"kubernetes.io/projected/7011a201-0f2a-490c-80fe-e9b5cea5398e-kube-api-access-ldxhk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.927796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.927759 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" event={"ID":"187669b7-aa6f-4124-8351-2dd33b9a19d6","Type":"ContainerStarted","Data":"a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f"} Oct 30 06:24:42.928013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.927841 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:42.929426 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.929390 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" event={"ID":"bed5e800-4476-4101-814f-27fa5caaba65","Type":"ContainerStarted","Data":"9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f"} Oct 30 06:24:42.929561 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.929437 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:24:42.931154 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.931131 5980 generic.go:356] "Generic (PLEG): container finished" podID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerID="6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325" exitCode=0 Oct 30 06:24:42.931232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.931163 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" Oct 30 06:24:42.931232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.931207 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" event={"ID":"7011a201-0f2a-490c-80fe-e9b5cea5398e","Type":"ContainerDied","Data":"6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325"} Oct 30 06:24:42.931314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.931253 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r" event={"ID":"7011a201-0f2a-490c-80fe-e9b5cea5398e","Type":"ContainerDied","Data":"f0738cb1a70952d7edcbae48c5301a08cbe5289d99ed9375a8d763866b4fa495"} Oct 30 06:24:42.931314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.931278 5980 scope.go:122] "RemoveContainer" containerID="6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325" Oct 30 06:24:42.942743 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.942726 5980 scope.go:122] "RemoveContainer" containerID="49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8" Oct 30 06:24:42.951403 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.951388 5980 scope.go:122] "RemoveContainer" containerID="6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325" Oct 30 06:24:42.951660 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:42.951639 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325\": container with ID starting with 6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325 not found: ID does not exist" containerID="6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325" Oct 30 06:24:42.951732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.951672 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325"} err="failed to get container status \"6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325\": rpc error: code = NotFound desc = could not find container \"6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325\": container with ID starting with 6a6e3b6122bb3d3272f2447880b45cdeba2e4a5d0fe37f9771be5a4ab1ab1325 not found: ID does not exist" Oct 30 06:24:42.951732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.951695 5980 scope.go:122] "RemoveContainer" containerID="49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8" Oct 30 06:24:42.951960 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:42.951945 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8\": container with ID starting with 49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8 not found: ID does not exist" containerID="49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8" Oct 30 06:24:42.952004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.951967 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8"} err="failed to get container status \"49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8\": rpc error: code = NotFound desc = could not find container \"49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8\": container with ID starting with 49ce2c5b8372c036cd29d362c67ddf51e80c999a3d6055b646fdbeffc0f5e4d8 not found: ID does not exist" Oct 30 06:24:42.976545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:42.976488 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" podStartSLOduration=2.158927945 podStartE2EDuration="3.976472175s" totalImagesPullingTime="1.81754423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:42.975063784 +0000 UTC m=+3137.083102819" watchObservedRunningTime="2026-10-30 06:24:42.976472175 +0000 UTC m=+3137.084511211" Oct 30 06:24:43.021277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:43.021236 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" podStartSLOduration=2.039143364 podStartE2EDuration="4.021223876s" totalImagesPullingTime="1.982080512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:43.020036839 +0000 UTC m=+3137.128075890" watchObservedRunningTime="2026-10-30 06:24:43.021223876 +0000 UTC m=+3137.129262909" Oct 30 06:24:43.095125 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:43.095079 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r"] Oct 30 06:24:43.098141 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:24:43.098068 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:24:43.107772 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:43.107746 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-9gb5r"] Oct 30 06:24:44.014212 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:44.014174 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" path="/var/lib/kubelet/pods/7011a201-0f2a-490c-80fe-e9b5cea5398e/volumes" Oct 30 06:24:53.940668 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:53.940632 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:24:53.941098 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:24:53.940705 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:25:11.907962 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:11.907926 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72"] Oct 30 06:25:11.908642 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:11.908222 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" containerID="cri-o://6590e91c78224afdf701fc5ff68e770926a08f624acf3e52d6264e4b7f2d89b2" gracePeriod=30 Oct 30 06:25:13.088336 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:13.088296 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:25:15.181706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:15.181659 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:25:15.182085 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:15.181738 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:25:17.122005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.121958 5980 generic.go:356] "Generic (PLEG): container finished" podID="8f345522-495a-42e2-9db8-d07dc01c289a" containerID="6590e91c78224afdf701fc5ff68e770926a08f624acf3e52d6264e4b7f2d89b2" exitCode=0 Oct 30 06:25:17.122503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.122168 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" event={"ID":"8f345522-495a-42e2-9db8-d07dc01c289a","Type":"ContainerDied","Data":"6590e91c78224afdf701fc5ff68e770926a08f624acf3e52d6264e4b7f2d89b2"} Oct 30 06:25:17.122503 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.122232 5980 scope.go:122] "RemoveContainer" containerID="d197ff44b82e29408bc3a65fb6cc1b8193028bb66fc9efac8316223fe0bff036" Oct 30 06:25:17.187926 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.187871 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 06:25:17.234173 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234148 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234204 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234309 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234275 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234325 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234428 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234425 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:17.234550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234478 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234550 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234523 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234567 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234620 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert\") pod \"8f345522-495a-42e2-9db8-d07dc01c289a\" (UID: \"8f345522-495a-42e2-9db8-d07dc01c289a\") " Oct 30 06:25:17.234720 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234685 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:17.234869 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.234827 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:17.235158 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235098 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:17.235278 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235179 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:17.235342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235314 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.235342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235335 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.235435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235350 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f345522-495a-42e2-9db8-d07dc01c289a-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.235435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235363 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.235435 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.235377 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.236699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.236671 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:17.236699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.236694 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "kube-api-access-s75kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:17.236882 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.236783 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:25:17.237448 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.237430 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy" pod "8f345522-495a-42e2-9db8-d07dc01c289a" (UID: "8f345522-495a-42e2-9db8-d07dc01c289a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:17.336141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.336096 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f345522-495a-42e2-9db8-d07dc01c289a-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.336141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.336136 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f345522-495a-42e2-9db8-d07dc01c289a-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.336141 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.336146 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75kl\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-kube-api-access-s75kl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.336618 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.336155 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f345522-495a-42e2-9db8-d07dc01c289a-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:17.398451 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:17.398421 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns"] Oct 30 06:25:18.128703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:18.128674 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" Oct 30 06:25:18.129167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:18.128724 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72" event={"ID":"8f345522-495a-42e2-9db8-d07dc01c289a","Type":"ContainerDied","Data":"4ccc44d949c3473e64d382665a54bbaa15aab9d151d45f2245b69fc666f52cb0"} Oct 30 06:25:18.129167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:18.128790 5980 scope.go:122] "RemoveContainer" containerID="6590e91c78224afdf701fc5ff68e770926a08f624acf3e52d6264e4b7f2d89b2" Oct 30 06:25:18.200958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:18.200918 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72"] Oct 30 06:25:18.214382 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:18.214355 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-8gw72"] Oct 30 06:25:20.012584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:20.012554 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" path="/var/lib/kubelet/pods/8f345522-495a-42e2-9db8-d07dc01c289a/volumes" Oct 30 06:25:32.418786 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.418729 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" containerID="cri-o://d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a" gracePeriod=45 Oct 30 06:25:32.683750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.683691 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 06:25:32.799650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.799613 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm\") pod \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " Oct 30 06:25:32.799840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.799714 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache\") pod \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " Oct 30 06:25:32.799840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.799812 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home\") pod \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " Oct 30 06:25:32.799987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.799841 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs\") pod \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " Oct 30 06:25:32.799987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.799873 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc\" (UniqueName: \"kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc\") pod \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " Oct 30 06:25:32.799987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.799909 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir\") pod \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\" (UID: \"42050220-36e0-4c55-a2e5-b4f1c66bebdf\") " Oct 30 06:25:32.800172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.800049 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache" pod "42050220-36e0-4c55-a2e5-b4f1c66bebdf" (UID: "42050220-36e0-4c55-a2e5-b4f1c66bebdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.800335 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.800311 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir" pod "42050220-36e0-4c55-a2e5-b4f1c66bebdf" (UID: "42050220-36e0-4c55-a2e5-b4f1c66bebdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.800537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.800389 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.800537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.800488 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home" pod "42050220-36e0-4c55-a2e5-b4f1c66bebdf" (UID: "42050220-36e0-4c55-a2e5-b4f1c66bebdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.803269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.803242 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm" pod "42050220-36e0-4c55-a2e5-b4f1c66bebdf" (UID: "42050220-36e0-4c55-a2e5-b4f1c66bebdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.803269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.803256 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs" pod "42050220-36e0-4c55-a2e5-b4f1c66bebdf" (UID: "42050220-36e0-4c55-a2e5-b4f1c66bebdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:32.803459 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.803253 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc" pod "42050220-36e0-4c55-a2e5-b4f1c66bebdf" (UID: "42050220-36e0-4c55-a2e5-b4f1c66bebdf"). InnerVolumeSpecName "kube-api-access-7dlwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:32.902078 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.902049 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.902078 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.902076 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.902288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.902089 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlwc\" (UniqueName: \"kubernetes.io/projected/42050220-36e0-4c55-a2e5-b4f1c66bebdf-kube-api-access-7dlwc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.902288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.902145 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.902288 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:32.902158 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42050220-36e0-4c55-a2e5-b4f1c66bebdf-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:33.203819 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.203785 5980 generic.go:356] "Generic (PLEG): container finished" podID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerID="d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a" exitCode=0 Oct 30 06:25:33.204010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.203848 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" Oct 30 06:25:33.204010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.203869 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" event={"ID":"42050220-36e0-4c55-a2e5-b4f1c66bebdf","Type":"ContainerDied","Data":"d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a"} Oct 30 06:25:33.204010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.203917 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns" event={"ID":"42050220-36e0-4c55-a2e5-b4f1c66bebdf","Type":"ContainerDied","Data":"75ad97176144c640cf4904049de10347d34e774a10497cf50db7393f83656ce2"} Oct 30 06:25:33.204010 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.203934 5980 scope.go:122] "RemoveContainer" containerID="d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a" Oct 30 06:25:33.213329 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.213310 5980 scope.go:122] "RemoveContainer" containerID="d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a" Oct 30 06:25:33.213620 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:33.213600 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a\": container with ID starting with d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a not found: ID does not exist" containerID="d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a" Oct 30 06:25:33.213703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.213629 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a"} err="failed to get container status \"d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a\": rpc error: code = NotFound desc = could not find container \"d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a\": container with ID starting with d3621fe2caa073ef4cafffd677b3f3f86985a4b0dfe077a76923d8891871a66a not found: ID does not exist" Oct 30 06:25:33.261473 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.261451 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns"] Oct 30 06:25:33.273980 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:33.273960 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/distinct-w4-e2e3d0-kserve-7d86656666-m5wns"] Oct 30 06:25:34.012978 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:34.012944 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" path="/var/lib/kubelet/pods/42050220-36e0-4c55-a2e5-b4f1c66bebdf/volumes" Oct 30 06:25:41.332028 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.331993 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6"] Oct 30 06:25:41.332471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.332221 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" podUID="187669b7-aa6f-4124-8351-2dd33b9a19d6" containerName="payload-processing" containerID="cri-o://a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f" gracePeriod=30 Oct 30 06:25:41.365081 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.364767 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2"] Oct 30 06:25:41.365368 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.365335 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" podUID="bed5e800-4476-4101-814f-27fa5caaba65" containerName="payload-pre-processing" containerID="cri-o://9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f" gracePeriod=30 Oct 30 06:25:41.689738 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.689712 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:25:41.706359 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.706330 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:25:41.786249 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786214 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume\") pod \"bed5e800-4476-4101-814f-27fa5caaba65\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " Oct 30 06:25:41.786447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786365 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d\" (UniqueName: \"kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d\") pod \"bed5e800-4476-4101-814f-27fa5caaba65\" (UID: \"bed5e800-4476-4101-814f-27fa5caaba65\") " Oct 30 06:25:41.786447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786425 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume\") pod \"187669b7-aa6f-4124-8351-2dd33b9a19d6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " Oct 30 06:25:41.786593 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786457 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz\" (UniqueName: \"kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz\") pod \"187669b7-aa6f-4124-8351-2dd33b9a19d6\" (UID: \"187669b7-aa6f-4124-8351-2dd33b9a19d6\") " Oct 30 06:25:41.786752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786723 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume" pod "bed5e800-4476-4101-814f-27fa5caaba65" (UID: "bed5e800-4476-4101-814f-27fa5caaba65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:41.786871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786798 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume" pod "187669b7-aa6f-4124-8351-2dd33b9a19d6" (UID: "187669b7-aa6f-4124-8351-2dd33b9a19d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:41.786937 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786925 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/187669b7-aa6f-4124-8351-2dd33b9a19d6-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:41.786998 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.786946 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bed5e800-4476-4101-814f-27fa5caaba65-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:41.788995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.788954 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz" pod "187669b7-aa6f-4124-8351-2dd33b9a19d6" (UID: "187669b7-aa6f-4124-8351-2dd33b9a19d6"). InnerVolumeSpecName "kube-api-access-w46jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:41.789153 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.789133 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d" pod "bed5e800-4476-4101-814f-27fa5caaba65" (UID: "bed5e800-4476-4101-814f-27fa5caaba65"). InnerVolumeSpecName "kube-api-access-d8m6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:41.887920 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.887840 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8m6d\" (UniqueName: \"kubernetes.io/projected/bed5e800-4476-4101-814f-27fa5caaba65-kube-api-access-d8m6d\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:41.887920 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:41.887868 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w46jz\" (UniqueName: \"kubernetes.io/projected/187669b7-aa6f-4124-8351-2dd33b9a19d6-kube-api-access-w46jz\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:42.249491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.249408 5980 generic.go:356] "Generic (PLEG): container finished" podID="187669b7-aa6f-4124-8351-2dd33b9a19d6" containerID="a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f" exitCode=0 Oct 30 06:25:42.249491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.249458 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" Oct 30 06:25:42.249705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.249493 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" event={"ID":"187669b7-aa6f-4124-8351-2dd33b9a19d6","Type":"ContainerDied","Data":"a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f"} Oct 30 06:25:42.249705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.249565 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6" event={"ID":"187669b7-aa6f-4124-8351-2dd33b9a19d6","Type":"ContainerDied","Data":"c43cb192e53cedbebb43583e77e211bbb9cf9c531ce55a8fef96e4b0bb9dff50"} Oct 30 06:25:42.249705 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.249601 5980 scope.go:122] "RemoveContainer" containerID="a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f" Oct 30 06:25:42.250898 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.250871 5980 generic.go:356] "Generic (PLEG): container finished" podID="bed5e800-4476-4101-814f-27fa5caaba65" containerID="9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f" exitCode=0 Oct 30 06:25:42.251013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.250912 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" Oct 30 06:25:42.251013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.250914 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" event={"ID":"bed5e800-4476-4101-814f-27fa5caaba65","Type":"ContainerDied","Data":"9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f"} Oct 30 06:25:42.251013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.250970 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2" event={"ID":"bed5e800-4476-4101-814f-27fa5caaba65","Type":"ContainerDied","Data":"67699499ae22424f275a157fc955e79bf785dd1f0e6d23d24c8f3611f3b105ed"} Oct 30 06:25:42.262656 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.262637 5980 scope.go:122] "RemoveContainer" containerID="a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f" Oct 30 06:25:42.262916 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:42.262896 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f\": container with ID starting with a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f not found: ID does not exist" containerID="a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f" Oct 30 06:25:42.262987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.262925 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f"} err="failed to get container status \"a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f\": rpc error: code = NotFound desc = could not find container \"a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f\": container with ID starting with a38d09bade40b6344c11172852fd73ad099b887d9ea40836e4b6d4b9e295da6f not found: ID does not exist" Oct 30 06:25:42.262987 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.262942 5980 scope.go:122] "RemoveContainer" containerID="9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f" Oct 30 06:25:42.271306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.271290 5980 scope.go:122] "RemoveContainer" containerID="9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f" Oct 30 06:25:42.271561 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:42.271544 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f\": container with ID starting with 9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f not found: ID does not exist" containerID="9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f" Oct 30 06:25:42.271615 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.271568 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f"} err="failed to get container status \"9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f\": rpc error: code = NotFound desc = could not find container \"9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f\": container with ID starting with 9fccab681a18aa490ae57438e47ad171f4bf575f847baaec20b2710efa49e72f not found: ID does not exist" Oct 30 06:25:42.310036 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.310010 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2"] Oct 30 06:25:42.323066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.323043 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-630fdf-7f6fc9cd79-4r7j2"] Oct 30 06:25:42.352542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.352515 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6"] Oct 30 06:25:42.355419 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:42.355402 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-630fdf-75fd9cc78-tlfg6"] Oct 30 06:25:43.087204 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:43.087167 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:25:44.012885 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:44.012851 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="187669b7-aa6f-4124-8351-2dd33b9a19d6" path="/var/lib/kubelet/pods/187669b7-aa6f-4124-8351-2dd33b9a19d6/volumes" Oct 30 06:25:44.013585 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:44.013397 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed5e800-4476-4101-814f-27fa5caaba65" path="/var/lib/kubelet/pods/bed5e800-4476-4101-814f-27fa5caaba65/volumes" Oct 30 06:25:47.633139 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.633091 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk"] Oct 30 06:25:47.633537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.633366 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" podUID="9e7fb19f-0864-4108-a51c-643f1d10fe16" containerName="maas-api" containerID="cri-o://bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a" gracePeriod=30 Oct 30 06:25:47.901746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.901714 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 06:25:47.955469 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.955442 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca\") pod \"9e7fb19f-0864-4108-a51c-643f1d10fe16\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " Oct 30 06:25:47.955644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.955481 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc\" (UniqueName: \"kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc\") pod \"9e7fb19f-0864-4108-a51c-643f1d10fe16\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " Oct 30 06:25:47.955644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.955562 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs\") pod \"9e7fb19f-0864-4108-a51c-643f1d10fe16\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " Oct 30 06:25:47.955644 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.955592 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls\") pod \"9e7fb19f-0864-4108-a51c-643f1d10fe16\" (UID: \"9e7fb19f-0864-4108-a51c-643f1d10fe16\") " Oct 30 06:25:47.956456 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.956425 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca" pod "9e7fb19f-0864-4108-a51c-643f1d10fe16" (UID: "9e7fb19f-0864-4108-a51c-643f1d10fe16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:47.958925 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.958885 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs" pod "9e7fb19f-0864-4108-a51c-643f1d10fe16" (UID: "9e7fb19f-0864-4108-a51c-643f1d10fe16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:47.959440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.959411 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc" pod "9e7fb19f-0864-4108-a51c-643f1d10fe16" (UID: "9e7fb19f-0864-4108-a51c-643f1d10fe16"). InnerVolumeSpecName "kube-api-access-sx9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:47.962620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:47.962588 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls" pod "9e7fb19f-0864-4108-a51c-643f1d10fe16" (UID: "9e7fb19f-0864-4108-a51c-643f1d10fe16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:48.056836 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.056807 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e7fb19f-0864-4108-a51c-643f1d10fe16-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.056836 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.056832 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9zc\" (UniqueName: \"kubernetes.io/projected/9e7fb19f-0864-4108-a51c-643f1d10fe16-kube-api-access-sx9zc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.056836 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.056842 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.057155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.056850 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e7fb19f-0864-4108-a51c-643f1d10fe16-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.286040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.285959 5980 generic.go:356] "Generic (PLEG): container finished" podID="9e7fb19f-0864-4108-a51c-643f1d10fe16" containerID="bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a" exitCode=0 Oct 30 06:25:48.286040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.286007 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" Oct 30 06:25:48.286313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.286042 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" event={"ID":"9e7fb19f-0864-4108-a51c-643f1d10fe16","Type":"ContainerDied","Data":"bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a"} Oct 30 06:25:48.286313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.286101 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk" event={"ID":"9e7fb19f-0864-4108-a51c-643f1d10fe16","Type":"ContainerDied","Data":"c871c5c3a9bc7ccba9c9d8fb6ed59cbae9bd2d85e2041fe12add8a900d3e576e"} Oct 30 06:25:48.286313 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.286140 5980 scope.go:122] "RemoveContainer" containerID="bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a" Oct 30 06:25:48.295940 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.295917 5980 scope.go:122] "RemoveContainer" containerID="bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a" Oct 30 06:25:48.296225 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:48.296205 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a\": container with ID starting with bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a not found: ID does not exist" containerID="bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a" Oct 30 06:25:48.296347 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.296236 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a"} err="failed to get container status \"bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a\": rpc error: code = NotFound desc = could not find container \"bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a\": container with ID starting with bb762e922ef994f2fcdbb506f35e693600660e04a8e57df13b81ce8231acd48a not found: ID does not exist" Oct 30 06:25:48.352497 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.352470 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk"] Oct 30 06:25:48.361884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:48.361859 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-e2e3d0-66c774896c-jcrjk"] Oct 30 06:25:50.013018 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:50.012986 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7fb19f-0864-4108-a51c-643f1d10fe16" path="/var/lib/kubelet/pods/9e7fb19f-0864-4108-a51c-643f1d10fe16/volumes" Oct 30 06:25:50.233941 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:50.233904 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs"] Oct 30 06:25:50.234162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:50.234139 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" containerID="cri-o://020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb" gracePeriod=30 Oct 30 06:25:55.490120 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.490077 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:25:55.636862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.636831 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637049 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.636875 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637049 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.636895 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637049 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637018 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637089 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637185 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637262 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637239 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637283 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637311 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket\") pod \"7d23e03c-e12c-4013-920c-e368e861d998\" (UID: \"7d23e03c-e12c-4013-920c-e368e861d998\") " Oct 30 06:25:55.637442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637342 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:55.637722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637681 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:55.637722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637710 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d23e03c-e12c-4013-920c-e368e861d998-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.638000 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637814 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:55.638000 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.637907 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:55.638127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.638022 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:55.639793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.639734 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:55.640088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.640052 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "kube-api-access-jf9ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:55.640088 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.640051 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:25:55.640247 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.640169 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token" pod "7d23e03c-e12c-4013-920c-e368e861d998" (UID: "7d23e03c-e12c-4013-920c-e368e861d998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:55.738187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738161 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9ld\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-kube-api-access-jf9ld\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738185 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738195 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d23e03c-e12c-4013-920c-e368e861d998-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738202 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738211 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738219 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d23e03c-e12c-4013-920c-e368e861d998-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738226 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.738341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:55.738234 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d23e03c-e12c-4013-920c-e368e861d998-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:25:56.332033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.331993 5980 generic.go:356] "Generic (PLEG): container finished" podID="7d23e03c-e12c-4013-920c-e368e861d998" containerID="020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb" exitCode=0 Oct 30 06:25:56.332217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.332040 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" Oct 30 06:25:56.332217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.332076 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" event={"ID":"7d23e03c-e12c-4013-920c-e368e861d998","Type":"ContainerDied","Data":"020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb"} Oct 30 06:25:56.332217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.332153 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs" event={"ID":"7d23e03c-e12c-4013-920c-e368e861d998","Type":"ContainerDied","Data":"5fcd390ea4a335c690cf5e462e9173cb48bd2fdc10265980b0ec910877365c7c"} Oct 30 06:25:56.332217 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.332178 5980 scope.go:122] "RemoveContainer" containerID="020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb" Oct 30 06:25:56.341199 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.341180 5980 scope.go:122] "RemoveContainer" containerID="23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941" Oct 30 06:25:56.349301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.349160 5980 scope.go:122] "RemoveContainer" containerID="020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb" Oct 30 06:25:56.349444 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:56.349425 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb\": container with ID starting with 020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb not found: ID does not exist" containerID="020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb" Oct 30 06:25:56.349489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.349455 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb"} err="failed to get container status \"020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb\": rpc error: code = NotFound desc = could not find container \"020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb\": container with ID starting with 020e1ceb0be2616c7f6e1a0ce6f2d1aaf377a89ba4f16cb9650425f763d247eb not found: ID does not exist" Oct 30 06:25:56.349489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.349476 5980 scope.go:122] "RemoveContainer" containerID="23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941" Oct 30 06:25:56.349710 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:25:56.349691 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941\": container with ID starting with 23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941 not found: ID does not exist" containerID="23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941" Oct 30 06:25:56.349751 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.349715 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941"} err="failed to get container status \"23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941\": rpc error: code = NotFound desc = could not find container \"23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941\": container with ID starting with 23fea2bbe7fec26bc58a1013f8826e3e9da78fbb7cddb6a1755ead73a9e0c941 not found: ID does not exist" Oct 30 06:25:56.410696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.409260 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs"] Oct 30 06:25:56.421586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:56.421563 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-630fdf-openshift-default-7d5dd78ff-g4cqs"] Oct 30 06:25:58.011955 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:58.011916 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d23e03c-e12c-4013-920c-e368e861d998" path="/var/lib/kubelet/pods/7d23e03c-e12c-4013-920c-e368e861d998/volumes" Oct 30 06:25:58.216017 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:58.215984 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5"] Oct 30 06:25:58.216267 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:58.216245 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" containerID="cri-o://b0a2feb369bd1e26df4cfb6a745822f0382115bd801f13477eb87729f2210149" gracePeriod=30 Oct 30 06:25:58.761938 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:58.761906 5980 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:25:58.762155 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:25:58.761962 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:26:01.144115 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:01.144081 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v"] Oct 30 06:26:03.370840 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.370801 5980 generic.go:356] "Generic (PLEG): container finished" podID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerID="b0a2feb369bd1e26df4cfb6a745822f0382115bd801f13477eb87729f2210149" exitCode=0 Oct 30 06:26:03.371344 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.370879 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" event={"ID":"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f","Type":"ContainerDied","Data":"b0a2feb369bd1e26df4cfb6a745822f0382115bd801f13477eb87729f2210149"} Oct 30 06:26:03.479004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.478978 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 06:26:03.611680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611654 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.611912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611753 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.611912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611780 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.611912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611822 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.611912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611869 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.611912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611900 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.612245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611928 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.612245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.611972 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.612245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612016 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data\") pod \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\" (UID: \"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f\") " Oct 30 06:26:03.612245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612154 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:03.612530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612359 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:03.612530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612280 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:03.612530 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612411 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.612750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612519 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:03.612750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.612567 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:03.614847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.614820 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:03.614966 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.614950 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:03.615061 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.615040 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:03.615252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.615231 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl" pod "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" (UID: "7eaf7b10-aed3-4bfa-a836-b7a51d037f5f"). InnerVolumeSpecName "kube-api-access-fj6gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:03.713465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713435 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713458 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713467 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713474 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713482 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj6gl\" (UniqueName: \"kubernetes.io/projected/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-kube-api-access-fj6gl\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713492 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713500 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:03.713713 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:03.713508 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:04.377854 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:04.377819 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" event={"ID":"7eaf7b10-aed3-4bfa-a836-b7a51d037f5f","Type":"ContainerDied","Data":"6690468f79f9be5263fc84f7bfb6113dffc93ed3ca076f340d0fcce624707328"} Oct 30 06:26:04.378333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:04.377873 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5" Oct 30 06:26:04.378333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:04.377881 5980 scope.go:122] "RemoveContainer" containerID="b0a2feb369bd1e26df4cfb6a745822f0382115bd801f13477eb87729f2210149" Oct 30 06:26:04.449260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:04.449237 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5"] Oct 30 06:26:04.462910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:04.462888 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-svrq5"] Oct 30 06:26:06.012539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:06.012505 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" path="/var/lib/kubelet/pods/7eaf7b10-aed3-4bfa-a836-b7a51d037f5f/volumes" Oct 30 06:26:06.145022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:06.144987 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerName="payload-processing" containerID="cri-o://41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642" gracePeriod=25 Oct 30 06:26:10.505393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:10.505359 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:13.086731 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:13.086687 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:26:13.646636 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:13.646603 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp"] Oct 30 06:26:13.646874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:13.646850 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" containerID="cri-o://fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2" gracePeriod=30 Oct 30 06:26:14.658033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:14.657998 5980 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:26:14.658416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:14.658054 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:26:18.907629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.907604 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:26:18.949122 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949049 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949122 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949091 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949135 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949158 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949179 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949225 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949281 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949311 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949357 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:18.949653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949367 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:18.949653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949363 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs\") pod \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\" (UID: \"1a254368-d09a-4e4a-a6e3-2f2e1446c64a\") " Oct 30 06:26:18.949653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949593 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:18.950047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949772 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:18.950047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949795 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:18.950047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949810 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:18.950047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949793 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:18.950047 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.949803 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:18.951568 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.951546 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:18.951568 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.951547 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:18.951725 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.951593 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:18.951725 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:18.951610 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q" pod "1a254368-d09a-4e4a-a6e3-2f2e1446c64a" (UID: "1a254368-d09a-4e4a-a6e3-2f2e1446c64a"). InnerVolumeSpecName "kube-api-access-bqx9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:19.050701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.050667 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.050701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.050692 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.050701 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.050701 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.050932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.050710 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqx9q\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-kube-api-access-bqx9q\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.050932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.050719 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.050932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.050727 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a254368-d09a-4e4a-a6e3-2f2e1446c64a-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.459641 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.459605 5980 generic.go:356] "Generic (PLEG): container finished" podID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerID="fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2" exitCode=0 Oct 30 06:26:19.459823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.459659 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" Oct 30 06:26:19.459823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.459661 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" event={"ID":"1a254368-d09a-4e4a-a6e3-2f2e1446c64a","Type":"ContainerDied","Data":"fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2"} Oct 30 06:26:19.459823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.459723 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp" event={"ID":"1a254368-d09a-4e4a-a6e3-2f2e1446c64a","Type":"ContainerDied","Data":"0babcfcd8e8875cb760fe3ecde547fb3308fbc8f787e17b99c3146c819200385"} Oct 30 06:26:19.459823 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.459748 5980 scope.go:122] "RemoveContainer" containerID="fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2" Oct 30 06:26:19.470769 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.470750 5980 scope.go:122] "RemoveContainer" containerID="2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4" Oct 30 06:26:19.480100 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.480085 5980 scope.go:122] "RemoveContainer" containerID="fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2" Oct 30 06:26:19.480378 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:19.480361 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2\": container with ID starting with fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2 not found: ID does not exist" containerID="fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2" Oct 30 06:26:19.480442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.480386 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2"} err="failed to get container status \"fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2\": rpc error: code = NotFound desc = could not find container \"fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2\": container with ID starting with fa9bce5c0879b080ab6b9a9bd81c9ded359b84b204cceb1784422834926343d2 not found: ID does not exist" Oct 30 06:26:19.480442 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.480403 5980 scope.go:122] "RemoveContainer" containerID="2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4" Oct 30 06:26:19.480650 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:19.480632 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4\": container with ID starting with 2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4 not found: ID does not exist" containerID="2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4" Oct 30 06:26:19.480700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.480656 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4"} err="failed to get container status \"2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4\": rpc error: code = NotFound desc = could not find container \"2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4\": container with ID starting with 2f70c28a24d9fbd9eee157d453ba4c4997971d0a3118735bd84156190e5e92e4 not found: ID does not exist" Oct 30 06:26:19.516183 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.516143 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp"] Oct 30 06:26:19.528996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:19.528969 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-trdmp"] Oct 30 06:26:20.012206 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:20.012172 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" path="/var/lib/kubelet/pods/1a254368-d09a-4e4a-a6e3-2f2e1446c64a/volumes" Oct 30 06:26:20.505364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:20.505321 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:24.690388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:24.690341 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6"] Oct 30 06:26:26.390091 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.390069 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:26:26.420965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.420907 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume\") pod \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " Oct 30 06:26:26.421077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.421033 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb\" (UniqueName: \"kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb\") pod \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\" (UID: \"f96ac33d-eca1-4daf-8e6f-f71c41c216d0\") " Oct 30 06:26:26.421274 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.421255 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume" pod "f96ac33d-eca1-4daf-8e6f-f71c41c216d0" (UID: "f96ac33d-eca1-4daf-8e6f-f71c41c216d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:26.421403 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.421382 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:26.423203 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.423182 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb" pod "f96ac33d-eca1-4daf-8e6f-f71c41c216d0" (UID: "f96ac33d-eca1-4daf-8e6f-f71c41c216d0"). InnerVolumeSpecName "kube-api-access-f7fdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:26.495014 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.494980 5980 generic.go:356] "Generic (PLEG): container finished" podID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerID="41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642" exitCode=0 Oct 30 06:26:26.495221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.495040 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" Oct 30 06:26:26.495221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.495080 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" event={"ID":"f96ac33d-eca1-4daf-8e6f-f71c41c216d0","Type":"ContainerDied","Data":"41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642"} Oct 30 06:26:26.495221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.495172 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v" event={"ID":"f96ac33d-eca1-4daf-8e6f-f71c41c216d0","Type":"ContainerDied","Data":"0f5d71541c13833c3ce0653cc4ac09504b67db600e304ccbfff93b26ca80fe90"} Oct 30 06:26:26.495221 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.495208 5980 scope.go:122] "RemoveContainer" containerID="41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642" Oct 30 06:26:26.506540 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.506513 5980 scope.go:122] "RemoveContainer" containerID="41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642" Oct 30 06:26:26.506870 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:26.506840 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642\": container with ID starting with 41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642 not found: ID does not exist" containerID="41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642" Oct 30 06:26:26.506938 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.506884 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642"} err="failed to get container status \"41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642\": rpc error: code = NotFound desc = could not find container \"41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642\": container with ID starting with 41dfe1371c8f3914aefd8623a66c56e8f2b978cd5e32b602c3d4e97ec5c31642 not found: ID does not exist" Oct 30 06:26:26.521997 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.521978 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7fdb\" (UniqueName: \"kubernetes.io/projected/f96ac33d-eca1-4daf-8e6f-f71c41c216d0-kube-api-access-f7fdb\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:26.550937 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.550913 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v"] Oct 30 06:26:26.557812 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:26.557786 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25d881-5b8cd7fd5c-hl46v"] Oct 30 06:26:28.012433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:28.012400 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" path="/var/lib/kubelet/pods/f96ac33d-eca1-4daf-8e6f-f71c41c216d0/volumes" Oct 30 06:26:29.691275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:29.691234 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerName="payload-pre-processing" containerID="cri-o://05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2" gracePeriod=25 Oct 30 06:26:34.021932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:34.021896 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn"] Oct 30 06:26:34.022431 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:34.022193 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" containerID="cri-o://1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f" gracePeriod=30 Oct 30 06:26:37.310911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:37.310833 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:38.168353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:38.168317 5980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:26:38.168531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:38.168370 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:26:39.270687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.270663 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:26:39.346071 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346045 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346130 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346170 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346287 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346197 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346298 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346328 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346369 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346411 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:39.346472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346449 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346517 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert\") pod \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\" (UID: \"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0\") " Oct 30 06:26:39.346724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346570 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:39.346903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346852 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.346903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.346874 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.347068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.347015 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:39.347068 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.347026 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:39.347218 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.347199 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:39.348732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.348708 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:39.348940 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.348917 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:39.349408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.349379 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:39.349408 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.349393 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq" pod "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" (UID: "9fc2dc0b-ba2c-4756-9190-d93b80a9cda0"). InnerVolumeSpecName "kube-api-access-7qrzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:39.447584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447557 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.447584 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447581 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.447732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447590 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.447732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447600 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.447732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447609 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qrzq\" (UniqueName: \"kubernetes.io/projected/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-kube-api-access-7qrzq\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.447732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447618 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.447732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.447626 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:39.558280 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.558250 5980 generic.go:356] "Generic (PLEG): container finished" podID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerID="1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f" exitCode=0 Oct 30 06:26:39.558406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.558307 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" Oct 30 06:26:39.558406 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.558301 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" event={"ID":"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0","Type":"ContainerDied","Data":"1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f"} Oct 30 06:26:39.558539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.558429 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn" event={"ID":"9fc2dc0b-ba2c-4756-9190-d93b80a9cda0","Type":"ContainerDied","Data":"c9e412d1d9a1e541e17a110bbf3678b414b10735fd36a2288b7a640e46ce8e78"} Oct 30 06:26:39.558539 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.558451 5980 scope.go:122] "RemoveContainer" containerID="1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f" Oct 30 06:26:39.568703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.568693 5980 scope.go:122] "RemoveContainer" containerID="8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e" Oct 30 06:26:39.577632 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.577616 5980 scope.go:122] "RemoveContainer" containerID="1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f" Oct 30 06:26:39.577871 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:39.577853 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f\": container with ID starting with 1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f not found: ID does not exist" containerID="1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f" Oct 30 06:26:39.577919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.577881 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f"} err="failed to get container status \"1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f\": rpc error: code = NotFound desc = could not find container \"1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f\": container with ID starting with 1691c55eeb689cf0141d0d805bae42896b8d2bfd413d1d1130ffb641f2e1ff3f not found: ID does not exist" Oct 30 06:26:39.577919 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.577899 5980 scope.go:122] "RemoveContainer" containerID="8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e" Oct 30 06:26:39.578146 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:39.578129 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e\": container with ID starting with 8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e not found: ID does not exist" containerID="8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e" Oct 30 06:26:39.578192 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.578151 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e"} err="failed to get container status \"8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e\": rpc error: code = NotFound desc = could not find container \"8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e\": container with ID starting with 8dd4acfff0e9804489b84fc8a0a32f2d971b9efc686347da6749285c5531fa8e not found: ID does not exist" Oct 30 06:26:39.627748 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.627697 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn"] Oct 30 06:26:39.642391 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:39.642367 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f6302c-openshift-default-775744b6b9-bfrdn"] Oct 30 06:26:40.012174 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:40.012084 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" path="/var/lib/kubelet/pods/9fc2dc0b-ba2c-4756-9190-d93b80a9cda0/volumes" Oct 30 06:26:43.086850 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:43.086818 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:26:47.310877 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:47.310844 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:49.935236 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:49.935206 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:26:50.037754 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.037680 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume\") pod \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " Oct 30 06:26:50.037918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.037784 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv\" (UniqueName: \"kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv\") pod \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\" (UID: \"1c651b7d-ee0a-43fe-a4e7-7ca798962171\") " Oct 30 06:26:50.038100 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.038077 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume" pod "1c651b7d-ee0a-43fe-a4e7-7ca798962171" (UID: "1c651b7d-ee0a-43fe-a4e7-7ca798962171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:50.039976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.039955 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv" pod "1c651b7d-ee0a-43fe-a4e7-7ca798962171" (UID: "1c651b7d-ee0a-43fe-a4e7-7ca798962171"). InnerVolumeSpecName "kube-api-access-hg9kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:50.139167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.139139 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c651b7d-ee0a-43fe-a4e7-7ca798962171-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.139167 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.139166 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg9kv\" (UniqueName: \"kubernetes.io/projected/1c651b7d-ee0a-43fe-a4e7-7ca798962171-kube-api-access-hg9kv\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.610264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.610232 5980 generic.go:356] "Generic (PLEG): container finished" podID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerID="05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2" exitCode=0 Oct 30 06:26:50.610475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.610288 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" Oct 30 06:26:50.610475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.610315 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" event={"ID":"1c651b7d-ee0a-43fe-a4e7-7ca798962171","Type":"ContainerDied","Data":"05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2"} Oct 30 06:26:50.610475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.610370 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6" event={"ID":"1c651b7d-ee0a-43fe-a4e7-7ca798962171","Type":"ContainerDied","Data":"2a37efcf7af34787eea39c8a018178dbd5644a2efeab0cbe85510f71fd4f63cc"} Oct 30 06:26:50.610475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.610391 5980 scope.go:122] "RemoveContainer" containerID="05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2" Oct 30 06:26:50.620369 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.620351 5980 scope.go:122] "RemoveContainer" containerID="05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2" Oct 30 06:26:50.620634 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:26:50.620617 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2\": container with ID starting with 05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2 not found: ID does not exist" containerID="05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2" Oct 30 06:26:50.620683 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.620645 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2"} err="failed to get container status \"05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2\": rpc error: code = NotFound desc = could not find container \"05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2\": container with ID starting with 05cea195b36f73bea33bd7de835715b0aae73c59ed223144909ad0ee9e7acaf2 not found: ID does not exist" Oct 30 06:26:50.667402 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.667381 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6"] Oct 30 06:26:50.680957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:50.680934 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f6302c-7ff9b899dc-mb4f6"] Oct 30 06:26:52.011056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:26:52.011020 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" path="/var/lib/kubelet/pods/1c651b7d-ee0a-43fe-a4e7-7ca798962171/volumes" Oct 30 06:27:00.620610 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.620575 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp"] Oct 30 06:27:00.621138 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.620794 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" podUID="7f875821-6c66-418b-b792-a749ad1e158a" containerName="maas-api" containerID="cri-o://82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b" gracePeriod=30 Oct 30 06:27:00.894783 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.894754 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 06:27:00.937688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.937658 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr\" (UniqueName: \"kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr\") pod \"7f875821-6c66-418b-b792-a749ad1e158a\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " Oct 30 06:27:00.937847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.937761 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls\") pod \"7f875821-6c66-418b-b792-a749ad1e158a\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " Oct 30 06:27:00.937847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.937825 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca\") pod \"7f875821-6c66-418b-b792-a749ad1e158a\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " Oct 30 06:27:00.937935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.937860 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs\") pod \"7f875821-6c66-418b-b792-a749ad1e158a\" (UID: \"7f875821-6c66-418b-b792-a749ad1e158a\") " Oct 30 06:27:00.938239 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.938212 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca" pod "7f875821-6c66-418b-b792-a749ad1e158a" (UID: "7f875821-6c66-418b-b792-a749ad1e158a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:00.940388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.940364 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs" pod "7f875821-6c66-418b-b792-a749ad1e158a" (UID: "7f875821-6c66-418b-b792-a749ad1e158a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:00.940520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.940398 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr" pod "7f875821-6c66-418b-b792-a749ad1e158a" (UID: "7f875821-6c66-418b-b792-a749ad1e158a"). InnerVolumeSpecName "kube-api-access-vn4pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:00.943149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:00.943092 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls" pod "7f875821-6c66-418b-b792-a749ad1e158a" (UID: "7f875821-6c66-418b-b792-a749ad1e158a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:01.039496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.039468 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:01.039496 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.039494 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f875821-6c66-418b-b792-a749ad1e158a-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:01.039719 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.039507 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f875821-6c66-418b-b792-a749ad1e158a-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:01.039719 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.039516 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn4pr\" (UniqueName: \"kubernetes.io/projected/7f875821-6c66-418b-b792-a749ad1e158a-kube-api-access-vn4pr\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:01.664011 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.663967 5980 generic.go:356] "Generic (PLEG): container finished" podID="7f875821-6c66-418b-b792-a749ad1e158a" containerID="82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b" exitCode=0 Oct 30 06:27:01.664553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.664027 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" Oct 30 06:27:01.664553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.664047 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" event={"ID":"7f875821-6c66-418b-b792-a749ad1e158a","Type":"ContainerDied","Data":"82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b"} Oct 30 06:27:01.664553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.664146 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp" event={"ID":"7f875821-6c66-418b-b792-a749ad1e158a","Type":"ContainerDied","Data":"4c5f363195e7b6a75b18bdae5f855b023628e579cad5e1a460356b99f5e68ac1"} Oct 30 06:27:01.664553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.664169 5980 scope.go:122] "RemoveContainer" containerID="82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b" Oct 30 06:27:01.674322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.674305 5980 scope.go:122] "RemoveContainer" containerID="82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b" Oct 30 06:27:01.674592 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:27:01.674573 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b\": container with ID starting with 82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b not found: ID does not exist" containerID="82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b" Oct 30 06:27:01.674659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.674604 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b"} err="failed to get container status \"82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b\": rpc error: code = NotFound desc = could not find container \"82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b\": container with ID starting with 82f7b614afb94df16608aa7a881b60df984f1b880424ea457bf816099e6c0e2b not found: ID does not exist" Oct 30 06:27:01.730055 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.730030 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp"] Oct 30 06:27:01.742246 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:01.742213 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c78066-5fb6dcf77b-phvbp"] Oct 30 06:27:02.011516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:02.011448 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f875821-6c66-418b-b792-a749ad1e158a" path="/var/lib/kubelet/pods/7f875821-6c66-418b-b792-a749ad1e158a/volumes" Oct 30 06:27:08.671868 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:08.671836 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552"] Oct 30 06:27:08.672379 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:08.672173 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" containerID="cri-o://040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe" gracePeriod=30 Oct 30 06:27:13.091097 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:27:13.088664 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:27:13.935673 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:13.935646 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 06:27:14.050646 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050582 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050646 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050619 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050646 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050639 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050664 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050714 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050740 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050771 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050819 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.050911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050845 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo\") pod \"e220f18d-16fd-406b-85d4-0f1250206231\" (UID: \"e220f18d-16fd-406b-85d4-0f1250206231\") " Oct 30 06:27:14.051277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.050966 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:14.051277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.051169 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:14.051277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.051247 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.051277 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.051267 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.051510 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.051409 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:14.051569 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.051515 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:14.051859 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.051834 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:14.053310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.053282 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:27:14.053438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.053387 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:14.053438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.053397 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "kube-api-access-rs84k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:14.053633 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.053621 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token" pod "e220f18d-16fd-406b-85d4-0f1250206231" (UID: "e220f18d-16fd-406b-85d4-0f1250206231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:14.151746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151719 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.151746 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151745 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.152189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151756 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs84k\" (UniqueName: \"kubernetes.io/projected/e220f18d-16fd-406b-85d4-0f1250206231-kube-api-access-rs84k\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.152189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151765 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e220f18d-16fd-406b-85d4-0f1250206231-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.152189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151774 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.152189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151794 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e220f18d-16fd-406b-85d4-0f1250206231-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.152189 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.151802 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e220f18d-16fd-406b-85d4-0f1250206231-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:27:14.727957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.727916 5980 generic.go:356] "Generic (PLEG): container finished" podID="e220f18d-16fd-406b-85d4-0f1250206231" containerID="040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe" exitCode=0 Oct 30 06:27:14.728164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.727981 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" Oct 30 06:27:14.728164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.728005 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" event={"ID":"e220f18d-16fd-406b-85d4-0f1250206231","Type":"ContainerDied","Data":"040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe"} Oct 30 06:27:14.728164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.728055 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552" event={"ID":"e220f18d-16fd-406b-85d4-0f1250206231","Type":"ContainerDied","Data":"00e3c2834b26808522fd58856320d0d4f3843000e7bfe4057a5335701d2ef481"} Oct 30 06:27:14.728164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.728078 5980 scope.go:122] "RemoveContainer" containerID="040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe" Oct 30 06:27:14.738121 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.738090 5980 scope.go:122] "RemoveContainer" containerID="040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe" Oct 30 06:27:14.738394 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:27:14.738374 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe\": container with ID starting with 040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe not found: ID does not exist" containerID="040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe" Oct 30 06:27:14.738451 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.738404 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe"} err="failed to get container status \"040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe\": rpc error: code = NotFound desc = could not find container \"040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe\": container with ID starting with 040122e75f443cfaa3cdeb5b6063b5a97d44094af0944e7fcaddfd77ff876ebe not found: ID does not exist" Oct 30 06:27:14.797724 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.797701 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552"] Oct 30 06:27:14.810971 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:14.810943 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-cr552"] Oct 30 06:27:16.011527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:16.011494 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e220f18d-16fd-406b-85d4-0f1250206231" path="/var/lib/kubelet/pods/e220f18d-16fd-406b-85d4-0f1250206231/volumes" Oct 30 06:27:24.366148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366097 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj"] Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366513 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366530 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366561 5980 cpu_manager.go:414] "containerMap: removing container" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerName="payload-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366568 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerName="payload-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366595 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366602 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366614 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366621 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366631 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366637 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366650 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366657 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366669 5980 cpu_manager.go:414] "containerMap: removing container" podUID="187669b7-aa6f-4124-8351-2dd33b9a19d6" containerName="payload-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366677 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="187669b7-aa6f-4124-8351-2dd33b9a19d6" containerName="payload-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366690 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366697 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366710 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366716 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366725 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366731 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366745 5980 cpu_manager.go:414] "containerMap: removing container" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366751 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366762 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7f875821-6c66-418b-b792-a749ad1e158a" containerName="maas-api" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366769 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f875821-6c66-418b-b792-a749ad1e158a" containerName="maas-api" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366779 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366785 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366800 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366807 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366823 5980 cpu_manager.go:414] "containerMap: removing container" podUID="9e7fb19f-0864-4108-a51c-643f1d10fe16" containerName="maas-api" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366830 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7fb19f-0864-4108-a51c-643f1d10fe16" containerName="maas-api" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366842 5980 cpu_manager.go:414] "containerMap: removing container" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366848 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366857 5980 cpu_manager.go:414] "containerMap: removing container" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerName="payload-pre-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366864 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerName="payload-pre-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366875 5980 cpu_manager.go:414] "containerMap: removing container" podUID="bed5e800-4476-4101-814f-27fa5caaba65" containerName="payload-pre-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366882 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed5e800-4476-4101-814f-27fa5caaba65" containerName="payload-pre-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366966 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366978 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366983 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366988 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366994 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc2dc0b-ba2c-4756-9190-d93b80a9cda0" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.366998 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="187669b7-aa6f-4124-8351-2dd33b9a19d6" containerName="payload-processing" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367004 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367011 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eaf7b10-aed3-4bfa-a836-b7a51d037f5f" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367017 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e220f18d-16fd-406b-85d4-0f1250206231" containerName="istio-proxy" Oct 30 06:27:24.366996 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367022 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="42050220-36e0-4c55-a2e5-b4f1c66bebdf" containerName="main" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367027 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f875821-6c66-418b-b792-a749ad1e158a" containerName="maas-api" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367032 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367037 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a254368-d09a-4e4a-a6e3-2f2e1446c64a" containerName="istio-proxy" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367043 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7fb19f-0864-4108-a51c-643f1d10fe16" containerName="maas-api" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367048 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c651b7d-ee0a-43fe-a4e7-7ca798962171" containerName="payload-pre-processing" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367054 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed5e800-4476-4101-814f-27fa5caaba65" containerName="payload-pre-processing" Oct 30 06:27:24.368691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.367059 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f96ac33d-eca1-4daf-8e6f-f71c41c216d0" containerName="payload-processing" Oct 30 06:27:24.371266 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.371248 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.403060 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.403035 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj"] Oct 30 06:27:24.442529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442497 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442529 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64aa156c-2329-46af-949b-482aa2562b42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442567 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442639 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64aa156c-2329-46af-949b-482aa2562b42-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442730 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442706 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64aa156c-2329-46af-949b-482aa2562b42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442741 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7gc\" (UniqueName: \"kubernetes.io/projected/64aa156c-2329-46af-949b-482aa2562b42-kube-api-access-hb7gc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442841 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442767 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442851 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.442911 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.442877 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543349 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543375 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543376 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64aa156c-2329-46af-949b-482aa2562b42-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543400 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64aa156c-2329-46af-949b-482aa2562b42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543433 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7gc\" (UniqueName: \"kubernetes.io/projected/64aa156c-2329-46af-949b-482aa2562b42-kube-api-access-hb7gc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543456 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543476 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543542 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543498 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543541 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543793 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543566 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64aa156c-2329-46af-949b-482aa2562b42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543805 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.543915 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543873 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.544025 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.543961 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.544103 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.544080 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.544553 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.544531 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64aa156c-2329-46af-949b-482aa2562b42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.545964 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.545941 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64aa156c-2329-46af-949b-482aa2562b42-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.546226 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.546208 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64aa156c-2329-46af-949b-482aa2562b42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.568357 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.568330 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64aa156c-2329-46af-949b-482aa2562b42-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.568482 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.568463 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7gc\" (UniqueName: \"kubernetes.io/projected/64aa156c-2329-46af-949b-482aa2562b42-kube-api-access-hb7gc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-w8wfj\" (UID: \"64aa156c-2329-46af-949b-482aa2562b42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:24.684659 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:24.684606 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:25.008944 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.008912 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj"] Oct 30 06:27:25.011291 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:27:25.011265 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64aa156c_2329_46af_949b_482aa2562b42.slice/crio-e85adcb8788c62cb24eb6e70272cef5f7679cf5b884aef120e8561142a5c0eba WatchSource:0}: Error finding container e85adcb8788c62cb24eb6e70272cef5f7679cf5b884aef120e8561142a5c0eba: Status 404 returned error can't find the container with id e85adcb8788c62cb24eb6e70272cef5f7679cf5b884aef120e8561142a5c0eba Oct 30 06:27:25.013520 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.013483 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:25.013635 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.013563 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:25.013635 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.013612 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:25.793013 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.792976 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" event={"ID":"64aa156c-2329-46af-949b-482aa2562b42","Type":"ContainerStarted","Data":"9fb00bb396e649603836d58ec436642dc145f1770354f9135882a44aa6b2fcbf"} Oct 30 06:27:25.793447 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.793026 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" event={"ID":"64aa156c-2329-46af-949b-482aa2562b42","Type":"ContainerStarted","Data":"e85adcb8788c62cb24eb6e70272cef5f7679cf5b884aef120e8561142a5c0eba"} Oct 30 06:27:25.855690 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:25.855642 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podStartSLOduration=1.85562943 podStartE2EDuration="1.85562943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:25.85532024 +0000 UTC m=+3299.963359272" watchObservedRunningTime="2026-10-30 06:27:25.85562943 +0000 UTC m=+3299.963668463" Oct 30 06:27:26.687136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:26.685380 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:26.688484 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:26.688460 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:27:26.688629 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:26.688512 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:27:27.687468 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:27.687435 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:27:27.687852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:27.687488 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:27:28.686490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:28.686451 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:27:28.686781 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:28.686747 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:27:29.687481 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:29.687440 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:27:29.687886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:29.687499 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:27:30.687486 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:30.687448 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:27:30.687897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:30.687513 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:27:31.685690 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:31.685657 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:27:31.685897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:31.685715 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:27:33.239943 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:33.239912 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:33.240489 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:33.240326 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:33.241178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:27:33.241157 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" Oct 30 06:27:43.086891 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:27:43.086856 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:28:13.086551 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:28:13.086513 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:28:39.548440 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.548402 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp"] Oct 30 06:28:39.549058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.548957 5980 cpu_manager.go:414] "containerMap: removing container" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" Oct 30 06:28:39.549058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.548977 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f345522-495a-42e2-9db8-d07dc01c289a" containerName="istio-proxy" Oct 30 06:28:39.549254 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.549157 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7011a201-0f2a-490c-80fe-e9b5cea5398e" containerName="istio-proxy" Oct 30 06:28:39.549254 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.549176 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d23e03c-e12c-4013-920c-e368e861d998" containerName="istio-proxy" Oct 30 06:28:39.556774 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.556740 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.592884 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.592853 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp"] Oct 30 06:28:39.665129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665078 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665140 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665220 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665265 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665304 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665337 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665324 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665422 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665465 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxph\" (UniqueName: \"kubernetes.io/projected/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-kube-api-access-8bxph\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.665620 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.665500 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766338 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766304 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766348 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766392 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766438 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxph\" (UniqueName: \"kubernetes.io/projected/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-kube-api-access-8bxph\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766480 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766537 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766563 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766590 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766616 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766799 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.766881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.766807 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.767146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.767125 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.767290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.767275 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.767733 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.767709 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.768821 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.768798 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.769311 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.769293 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.791417 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.791388 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.791535 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.791503 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxph\" (UniqueName: \"kubernetes.io/projected/499c0ecb-e7fc-47e7-9435-9be34a40ecb3-kube-api-access-8bxph\") pod \"maas-default-gateway-openshift-default-58847f9fc5-74tmp\" (UID: \"499c0ecb-e7fc-47e7-9435-9be34a40ecb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:39.873027 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:39.873003 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:40.190855 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:40.190829 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp"] Oct 30 06:28:40.193017 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:28:40.192981 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499c0ecb_e7fc_47e7_9435_9be34a40ecb3.slice/crio-de3969bab84d499cf5cd50c37d13e9ff2cd0af8613746da8192a07e2a7a552a7 WatchSource:0}: Error finding container de3969bab84d499cf5cd50c37d13e9ff2cd0af8613746da8192a07e2a7a552a7: Status 404 returned error can't find the container with id de3969bab84d499cf5cd50c37d13e9ff2cd0af8613746da8192a07e2a7a552a7 Oct 30 06:28:40.195493 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:40.195460 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:40.195586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:40.195544 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:40.195586 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:40.195572 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:41.117718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:41.117680 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" event={"ID":"499c0ecb-e7fc-47e7-9435-9be34a40ecb3","Type":"ContainerStarted","Data":"01e059b61c26a8358698d5a6acb0ff1e15c7e3d41ce836bf42a0453f79cdeee0"} Oct 30 06:28:41.118253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:41.117745 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" event={"ID":"499c0ecb-e7fc-47e7-9435-9be34a40ecb3","Type":"ContainerStarted","Data":"de3969bab84d499cf5cd50c37d13e9ff2cd0af8613746da8192a07e2a7a552a7"} Oct 30 06:28:41.884132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:41.880651 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:41.886714 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:41.886688 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:41.886897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:41.886872 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:42.876488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:42.876455 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:42.877083 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:42.877052 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:43.108993 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:28:43.108953 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:28:43.878734 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:43.878703 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:43.879341 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:43.879307 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:44.873529 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:44.873491 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:44.873979 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:44.873947 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:45.878492 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:45.878453 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:45.879094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:45.879061 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:46.873926 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:46.873889 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:46.874243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:46.874211 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:47.873974 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:47.873937 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 06:28:47.874500 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:47.874012 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 06:28:49.915306 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:49.915267 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-74tmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:28:49.915768 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:49.915340 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podUID="499c0ecb-e7fc-47e7-9435-9be34a40ecb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:28:50.697588 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:50.697555 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:50.698012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:50.697995 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:50.708241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:50.708222 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" Oct 30 06:28:50.759463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:28:50.759422 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-74tmp" podStartSLOduration=11.759410506 podStartE2EDuration="11.759410506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:41.188683655 +0000 UTC m=+3375.296722689" watchObservedRunningTime="2026-10-30 06:28:50.759410506 +0000 UTC m=+3384.867449539" Oct 30 06:29:04.284591 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:29:04.284204 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:29:04.284591 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:29:04.284279 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:29:13.086622 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:29:13.086579 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:29:43.086119 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:29:43.086013 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:30:13.087386 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:30:13.087344 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:30:28.950121 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:30:28.950071 5980 scope.go:122] "RemoveContainer" containerID="f908b57b043bab5a872e211e71bcc17e9b45cab98b3f25d90bcd327cf7363882" Oct 30 06:30:43.086324 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:30:43.086285 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:31:13.085861 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:31:13.085828 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:31:43.085601 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:31:43.085567 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:32:13.086356 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:32:13.086318 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:32:23.252499 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.252462 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9"] Oct 30 06:32:23.256276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.256257 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.278917 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.278883 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-6c0c09-openshift-default-dockercfg-2kffc\"" Oct 30 06:32:23.290594 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.290571 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9"] Oct 30 06:32:23.397628 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.397594 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.397826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.397632 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.397826 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.397667 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.397949 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.397819 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.397949 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.397880 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.398066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.397976 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.398066 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.398039 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5vk\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.398224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.398077 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.398224 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.398189 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.500935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.499445 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.500935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.499985 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.500935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.500016 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.500935 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.500062 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.500096 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501001 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501018 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.499884 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501095 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501195 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5vk\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501234 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501343 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501298 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501491 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.501796 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.501572 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.503861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.503798 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.504004 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.503970 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.525092 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.525061 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.525537 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.525518 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5vk\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk\") pod \"e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.570360 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.570340 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:23.894809 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.894779 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9"] Oct 30 06:32:23.896576 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:32:23.896548 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cd83bf_9685_4e52_a6fe_088b24207262.slice/crio-6c7088e9b4f88a73c1580b08d5534b3719592cc5c02ef5e0091afb5d9553fc9a WatchSource:0}: Error finding container 6c7088e9b4f88a73c1580b08d5534b3719592cc5c02ef5e0091afb5d9553fc9a: Status 404 returned error can't find the container with id 6c7088e9b4f88a73c1580b08d5534b3719592cc5c02ef5e0091afb5d9553fc9a Oct 30 06:32:23.898912 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.898874 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:23.898972 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.898957 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:23.899019 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:23.899004 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:24.239241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:24.239143 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" event={"ID":"46cd83bf-9685-4e52-a6fe-088b24207262","Type":"ContainerStarted","Data":"6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492"} Oct 30 06:32:24.239241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:24.239201 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" event={"ID":"46cd83bf-9685-4e52-a6fe-088b24207262","Type":"ContainerStarted","Data":"6c7088e9b4f88a73c1580b08d5534b3719592cc5c02ef5e0091afb5d9553fc9a"} Oct 30 06:32:24.288479 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:24.288432 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podStartSLOduration=1.288420018 podStartE2EDuration="1.288420018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:24.287481219 +0000 UTC m=+3598.395520252" watchObservedRunningTime="2026-10-30 06:32:24.288420018 +0000 UTC m=+3598.396459051" Oct 30 06:32:24.571147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:24.571095 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:24.575993 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:24.575972 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:25.243622 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:25.243591 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:25.244661 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:25.244644 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:28.546557 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.546521 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4"] Oct 30 06:32:28.549847 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.549827 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.568697 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.568673 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6c0c09\"" Oct 30 06:32:28.573891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.573876 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-73b62168-dockercfg-bw9gt\"" Oct 30 06:32:28.588693 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.588668 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4"] Oct 30 06:32:28.653204 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.653169 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjfcx\" (UniqueName: \"kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx\") pod \"payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.653566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.653537 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.754433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.754395 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.754631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.754495 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjfcx\" (UniqueName: \"kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx\") pod \"payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.755117 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.755081 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.779391 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.779363 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjfcx\" (UniqueName: \"kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx\") pod \"payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:28.862164 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:28.862134 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:29.169598 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:29.169572 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4"] Oct 30 06:32:29.171676 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:32:29.171649 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77627e7b_a612_4eac_b8b0_1e7e10b9e79b.slice/crio-01fe4018438f6c713c7f77789a393eccae22d53273000801df5117017e06d2aa WatchSource:0}: Error finding container 01fe4018438f6c713c7f77789a393eccae22d53273000801df5117017e06d2aa: Status 404 returned error can't find the container with id 01fe4018438f6c713c7f77789a393eccae22d53273000801df5117017e06d2aa Oct 30 06:32:29.263634 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:29.263602 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" event={"ID":"77627e7b-a612-4eac-b8b0-1e7e10b9e79b","Type":"ContainerStarted","Data":"9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d"} Oct 30 06:32:29.263779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:29.263679 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" event={"ID":"77627e7b-a612-4eac-b8b0-1e7e10b9e79b","Type":"ContainerStarted","Data":"01fe4018438f6c713c7f77789a393eccae22d53273000801df5117017e06d2aa"} Oct 30 06:32:29.263779 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:29.263736 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:29.307785 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:29.307736 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" podStartSLOduration=1.307723662 podStartE2EDuration="1.307723662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:29.3072092 +0000 UTC m=+3603.415248238" watchObservedRunningTime="2026-10-30 06:32:29.307723662 +0000 UTC m=+3603.415762695" Oct 30 06:32:40.271675 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:40.271641 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:32:41.284245 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:41.284205 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:32:41.285600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:41.284272 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:32:42.334627 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:42.334595 5980 generic.go:356] "Generic (PLEG): container finished" podID="46cd83bf-9685-4e52-a6fe-088b24207262" containerID="6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492" exitCode=0 Oct 30 06:32:42.335021 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:42.334670 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" event={"ID":"46cd83bf-9685-4e52-a6fe-088b24207262","Type":"ContainerDied","Data":"6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492"} Oct 30 06:32:42.335067 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:42.335055 5980 scope.go:122] "RemoveContainer" containerID="6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492" Oct 30 06:32:42.336259 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:42.336224 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:42.336330 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:42.336291 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:42.336330 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:42.336322 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:32:43.109525 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:32:43.109486 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:32:43.340948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:43.340916 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" event={"ID":"46cd83bf-9685-4e52-a6fe-088b24207262","Type":"ContainerStarted","Data":"f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff"} Oct 30 06:32:43.587135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:43.584505 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:43.587135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:43.584634 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:32:43.587135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:43.584678 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 06:32:44.589264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:44.586537 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:32:44.589264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:44.586632 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 06:32:45.573507 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:45.573473 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:32:45.574230 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:45.573796 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 06:32:46.581132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:46.579495 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:32:46.581132 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:46.579558 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 06:32:47.578135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:47.574429 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 06:32:47.578135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:47.574486 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 06:32:48.893497 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:48.893461 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:48.894051 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:48.893990 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:32:48.895134 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:32:48.895087 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:33:13.086966 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:33:13.086932 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:33:29.085077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:29.085042 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4"] Oct 30 06:33:34.085485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:34.085430 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerName="payload-pre-processing" containerID="cri-o://9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d" gracePeriod=25 Oct 30 06:33:38.124194 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:38.124155 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9"] Oct 30 06:33:38.124703 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:38.124475 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" containerID="cri-o://f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff" gracePeriod=30 Oct 30 06:33:40.270547 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:40.270510 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:33:43.086601 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:33:43.086569 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:33:43.382034 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.382012 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:33:43.474676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474649 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.474832 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474694 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.474832 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474738 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.474832 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474781 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.475005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474833 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.475005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474882 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.475005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474913 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.475005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474953 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.475005 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.474996 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket\") pod \"46cd83bf-9685-4e52-a6fe-088b24207262\" (UID: \"46cd83bf-9685-4e52-a6fe-088b24207262\") " Oct 30 06:33:43.475948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.475225 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:43.475948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.475390 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:43.475948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.475398 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.475948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.475563 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:43.475948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.475764 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:43.475948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.475780 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:43.477241 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.477216 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "kube-api-access-zr5vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:43.477352 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.477293 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:33:43.477480 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.477461 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:43.478205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.478184 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy" pod "46cd83bf-9685-4e52-a6fe-088b24207262" (UID: "46cd83bf-9685-4e52-a6fe-088b24207262"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:33:43.576260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576237 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576257 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr5vk\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-kube-api-access-zr5vk\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576267 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576276 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd83bf-9685-4e52-a6fe-088b24207262-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576285 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576292 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd83bf-9685-4e52-a6fe-088b24207262-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576300 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd83bf-9685-4e52-a6fe-088b24207262-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.576394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.576307 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd83bf-9685-4e52-a6fe-088b24207262-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.621373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.621317 5980 generic.go:356] "Generic (PLEG): container finished" podID="46cd83bf-9685-4e52-a6fe-088b24207262" containerID="f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff" exitCode=0 Oct 30 06:33:43.621373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.621367 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" Oct 30 06:33:43.621511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.621390 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" event={"ID":"46cd83bf-9685-4e52-a6fe-088b24207262","Type":"ContainerDied","Data":"f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff"} Oct 30 06:33:43.621511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.621458 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9" event={"ID":"46cd83bf-9685-4e52-a6fe-088b24207262","Type":"ContainerDied","Data":"6c7088e9b4f88a73c1580b08d5534b3719592cc5c02ef5e0091afb5d9553fc9a"} Oct 30 06:33:43.621511 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.621481 5980 scope.go:122] "RemoveContainer" containerID="f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff" Oct 30 06:33:43.631514 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.631497 5980 scope.go:122] "RemoveContainer" containerID="6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492" Oct 30 06:33:43.642597 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.642582 5980 scope.go:122] "RemoveContainer" containerID="f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff" Oct 30 06:33:43.642927 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:33:43.642905 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff\": container with ID starting with f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff not found: ID does not exist" containerID="f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff" Oct 30 06:33:43.643016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.642937 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff"} err="failed to get container status \"f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff\": rpc error: code = NotFound desc = could not find container \"f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff\": container with ID starting with f612185f40d17047582fadfbefb0cb60b5cb54d10bd5bd8256583a6124ad59ff not found: ID does not exist" Oct 30 06:33:43.643016 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.642961 5980 scope.go:122] "RemoveContainer" containerID="6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492" Oct 30 06:33:43.643274 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:33:43.643254 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492\": container with ID starting with 6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492 not found: ID does not exist" containerID="6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492" Oct 30 06:33:43.643322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.643280 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492"} err="failed to get container status \"6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492\": rpc error: code = NotFound desc = could not find container \"6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492\": container with ID starting with 6e48c603be204d2f930d0386b4b1789d476e3a6f3908e9ac37e0a1fe113a5492 not found: ID does not exist" Oct 30 06:33:43.678150 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.678129 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9"] Oct 30 06:33:43.684040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:43.684017 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-6c0c09-openshift-default-5c656554c7wtrt9"] Oct 30 06:33:44.011300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:44.011224 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" path="/var/lib/kubelet/pods/46cd83bf-9685-4e52-a6fe-088b24207262/volumes" Oct 30 06:33:50.270409 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:50.270373 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:33:54.336315 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.336293 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:33:54.500441 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.500355 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx\" (UniqueName: \"kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx\") pod \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " Oct 30 06:33:54.500441 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.500392 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume\") pod \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\" (UID: \"77627e7b-a612-4eac-b8b0-1e7e10b9e79b\") " Oct 30 06:33:54.500861 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.500833 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume" pod "77627e7b-a612-4eac-b8b0-1e7e10b9e79b" (UID: "77627e7b-a612-4eac-b8b0-1e7e10b9e79b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:54.502932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.502909 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx" pod "77627e7b-a612-4eac-b8b0-1e7e10b9e79b" (UID: "77627e7b-a612-4eac-b8b0-1e7e10b9e79b"). InnerVolumeSpecName "kube-api-access-fjfcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:54.601845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.601815 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjfcx\" (UniqueName: \"kubernetes.io/projected/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-kube-api-access-fjfcx\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:54.601845 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.601842 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77627e7b-a612-4eac-b8b0-1e7e10b9e79b-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:33:54.679592 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.679555 5980 generic.go:356] "Generic (PLEG): container finished" podID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerID="9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d" exitCode=0 Oct 30 06:33:54.679747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.679604 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" Oct 30 06:33:54.679747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.679604 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" event={"ID":"77627e7b-a612-4eac-b8b0-1e7e10b9e79b","Type":"ContainerDied","Data":"9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d"} Oct 30 06:33:54.679747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.679660 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4" event={"ID":"77627e7b-a612-4eac-b8b0-1e7e10b9e79b","Type":"ContainerDied","Data":"01fe4018438f6c713c7f77789a393eccae22d53273000801df5117017e06d2aa"} Oct 30 06:33:54.679747 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.679687 5980 scope.go:122] "RemoveContainer" containerID="9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d" Oct 30 06:33:54.688910 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.688886 5980 scope.go:122] "RemoveContainer" containerID="9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d" Oct 30 06:33:54.689195 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:33:54.689174 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d\": container with ID starting with 9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d not found: ID does not exist" containerID="9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d" Oct 30 06:33:54.689297 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.689202 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d"} err="failed to get container status \"9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d\": rpc error: code = NotFound desc = could not find container \"9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d\": container with ID starting with 9dadff981148571827475933875221c76ce4212420c98361bd30aa3187ad2c5d not found: ID does not exist" Oct 30 06:33:54.734750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.734723 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4"] Oct 30 06:33:54.748282 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:54.748257 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6c0c09-d8c66cw5b4"] Oct 30 06:33:56.011652 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:33:56.011614 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" path="/var/lib/kubelet/pods/77627e7b-a612-4eac-b8b0-1e7e10b9e79b/volumes" Oct 30 06:34:01.662886 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:01.662853 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24"] Oct 30 06:34:01.663333 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:01.663068 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" podUID="03b4ebd4-0abd-4b74-a016-900c456bf9e5" containerName="manager" containerID="cri-o://c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d" gracePeriod=10 Oct 30 06:34:01.919472 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:01.919414 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 06:34:02.070715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.070687 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh\" (UniqueName: \"kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh\") pod \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " Oct 30 06:34:02.070867 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.070765 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume\") pod \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\" (UID: \"03b4ebd4-0abd-4b74-a016-900c456bf9e5\") " Oct 30 06:34:02.071171 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.071148 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume" pod "03b4ebd4-0abd-4b74-a016-900c456bf9e5" (UID: "03b4ebd4-0abd-4b74-a016-900c456bf9e5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:34:02.072971 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.072947 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh" pod "03b4ebd4-0abd-4b74-a016-900c456bf9e5" (UID: "03b4ebd4-0abd-4b74-a016-900c456bf9e5"). InnerVolumeSpecName "kube-api-access-ldnbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:34:02.171619 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.171558 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldnbh\" (UniqueName: \"kubernetes.io/projected/03b4ebd4-0abd-4b74-a016-900c456bf9e5-kube-api-access-ldnbh\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:34:02.171619 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.171581 5980 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/03b4ebd4-0abd-4b74-a016-900c456bf9e5-extensions-socket-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:34:02.724264 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.724233 5980 generic.go:356] "Generic (PLEG): container finished" podID="03b4ebd4-0abd-4b74-a016-900c456bf9e5" containerID="c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d" exitCode=0 Oct 30 06:34:02.724680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.724278 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" Oct 30 06:34:02.724680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.724305 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" event={"ID":"03b4ebd4-0abd-4b74-a016-900c456bf9e5","Type":"ContainerDied","Data":"c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d"} Oct 30 06:34:02.724680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.724346 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24" event={"ID":"03b4ebd4-0abd-4b74-a016-900c456bf9e5","Type":"ContainerDied","Data":"c7aa7cfc160301b88d903081e2fb3a20815ad96b8b54d84f860a3b79785b00b2"} Oct 30 06:34:02.724680 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.724366 5980 scope.go:122] "RemoveContainer" containerID="c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d" Oct 30 06:34:02.733692 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.733676 5980 scope.go:122] "RemoveContainer" containerID="c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d" Oct 30 06:34:02.733946 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:34:02.733929 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d\": container with ID starting with c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d not found: ID does not exist" containerID="c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d" Oct 30 06:34:02.733995 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.733956 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d"} err="failed to get container status \"c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d\": rpc error: code = NotFound desc = could not find container \"c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d\": container with ID starting with c0250582e0751bb917629e6a09b475557820b87a7e27494fdc703f2f288b877d not found: ID does not exist" Oct 30 06:34:02.793428 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.793401 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24"] Oct 30 06:34:02.813415 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:02.808774 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-m6j24"] Oct 30 06:34:04.011907 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:34:04.011860 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b4ebd4-0abd-4b74-a016-900c456bf9e5" path="/var/lib/kubelet/pods/03b4ebd4-0abd-4b74-a016-900c456bf9e5/volumes" Oct 30 06:34:13.086087 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:34:13.086052 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:34:43.086070 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:34:43.086035 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:35:04.856871 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.856839 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7"] Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857216 5980 cpu_manager.go:414] "containerMap: removing container" podUID="03b4ebd4-0abd-4b74-a016-900c456bf9e5" containerName="manager" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857229 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b4ebd4-0abd-4b74-a016-900c456bf9e5" containerName="manager" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857239 5980 cpu_manager.go:414] "containerMap: removing container" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857244 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857274 5980 cpu_manager.go:414] "containerMap: removing container" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerName="payload-pre-processing" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857280 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerName="payload-pre-processing" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857338 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="77627e7b-a612-4eac-b8b0-1e7e10b9e79b" containerName="payload-pre-processing" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857345 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" Oct 30 06:35:04.857390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.857353 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b4ebd4-0abd-4b74-a016-900c456bf9e5" containerName="manager" Oct 30 06:35:04.860485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.860465 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:04.879773 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.879752 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7"] Oct 30 06:35:04.892163 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.892144 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n9wqs\"" Oct 30 06:35:04.921914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.921892 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b072765-4f9f-4379-b544-ecc9650203c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rjtn7\" (UID: \"6b072765-4f9f-4379-b544-ecc9650203c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:04.922065 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:04.921962 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcs8\" (UniqueName: \"kubernetes.io/projected/6b072765-4f9f-4379-b544-ecc9650203c3-kube-api-access-mhcs8\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rjtn7\" (UID: \"6b072765-4f9f-4379-b544-ecc9650203c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:05.023283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.023253 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcs8\" (UniqueName: \"kubernetes.io/projected/6b072765-4f9f-4379-b544-ecc9650203c3-kube-api-access-mhcs8\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rjtn7\" (UID: \"6b072765-4f9f-4379-b544-ecc9650203c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:05.023476 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.023332 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b072765-4f9f-4379-b544-ecc9650203c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rjtn7\" (UID: \"6b072765-4f9f-4379-b544-ecc9650203c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:05.023729 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.023709 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6b072765-4f9f-4379-b544-ecc9650203c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rjtn7\" (UID: \"6b072765-4f9f-4379-b544-ecc9650203c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:05.048154 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.048126 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcs8\" (UniqueName: \"kubernetes.io/projected/6b072765-4f9f-4379-b544-ecc9650203c3-kube-api-access-mhcs8\") pod \"kuadrant-operator-controller-manager-57bb9bfff-rjtn7\" (UID: \"6b072765-4f9f-4379-b544-ecc9650203c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:05.173322 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.173253 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:05.502128 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.502086 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7"] Oct 30 06:35:05.504582 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:35:05.504554 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b072765_4f9f_4379_b544_ecc9650203c3.slice/crio-cc1beb37b2928d9cb644d08c2739fba6a77a65bb7fd4c91832aeb64e2cb32a98 WatchSource:0}: Error finding container cc1beb37b2928d9cb644d08c2739fba6a77a65bb7fd4c91832aeb64e2cb32a98: Status 404 returned error can't find the container with id cc1beb37b2928d9cb644d08c2739fba6a77a65bb7fd4c91832aeb64e2cb32a98 Oct 30 06:35:05.983398 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.983361 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" event={"ID":"6b072765-4f9f-4379-b544-ecc9650203c3","Type":"ContainerStarted","Data":"cdfeb29a090a5b30ea266dc10d5c89704192808013c955bf1bb511134f5aeae0"} Oct 30 06:35:05.983780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.983436 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" event={"ID":"6b072765-4f9f-4379-b544-ecc9650203c3","Type":"ContainerStarted","Data":"cc1beb37b2928d9cb644d08c2739fba6a77a65bb7fd4c91832aeb64e2cb32a98"} Oct 30 06:35:05.983780 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:05.983515 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:06.031957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:06.031913 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" podStartSLOduration=2.031901439 podStartE2EDuration="2.031901439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:06.030741779 +0000 UTC m=+3760.138780811" watchObservedRunningTime="2026-10-30 06:35:06.031901439 +0000 UTC m=+3760.139940473" Oct 30 06:35:13.086504 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:35:13.086463 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:35:16.989830 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:16.989801 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-rjtn7" Oct 30 06:35:43.153282 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:35:43.153183 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:35:44.019146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:44.018584 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:35:44.019146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:44.018641 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:56.386516 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.386484 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq"] Oct 30 06:35:56.386901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.386841 5980 cpu_manager.go:414] "containerMap: removing container" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" Oct 30 06:35:56.386901 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.386852 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" Oct 30 06:35:56.386976 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.386943 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cd83bf-9685-4e52-a6fe-088b24207262" containerName="istio-proxy" Oct 30 06:35:56.390296 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.390275 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.406581 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.406559 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ee1c63-openshift-default-dockercfg-vt65x\"" Oct 30 06:35:56.422136 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.422084 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq"] Oct 30 06:35:56.527001 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.526962 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527014 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527235 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527189 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527332 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527261 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527332 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527291 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgp4\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527404 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527370 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527465 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527446 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527504 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527482 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.527571 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.527559 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628145 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628178 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628179 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgp4\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628201 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628248 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628271 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628297 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628321 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628360 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628438 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628411 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628732 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628801 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628769 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628835 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.628924 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.628862 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.629250 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.629234 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.630909 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.630893 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.631197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.631171 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.653077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.653018 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgp4\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.653077 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.653034 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token\") pod \"e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:56.703033 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:56.703011 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:57.028146 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.025495 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq"] Oct 30 06:35:57.031339 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:35:57.031314 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351ac9ed_73a7_49cc_b4f8_acaff83228eb.slice/crio-c25f0c80629722279a0cfa2057b88e33993da69e3485fa5ed43f1c4d2ce3854e WatchSource:0}: Error finding container c25f0c80629722279a0cfa2057b88e33993da69e3485fa5ed43f1c4d2ce3854e: Status 404 returned error can't find the container with id c25f0c80629722279a0cfa2057b88e33993da69e3485fa5ed43f1c4d2ce3854e Oct 30 06:35:57.033365 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.033327 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:57.033445 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.033409 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:57.033490 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.033452 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:57.201992 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.201954 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" event={"ID":"351ac9ed-73a7-49cc-b4f8-acaff83228eb","Type":"ContainerStarted","Data":"8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2"} Oct 30 06:35:57.202205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.202022 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" event={"ID":"351ac9ed-73a7-49cc-b4f8-acaff83228eb","Type":"ContainerStarted","Data":"c25f0c80629722279a0cfa2057b88e33993da69e3485fa5ed43f1c4d2ce3854e"} Oct 30 06:35:57.250750 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.250698 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podStartSLOduration=1.2506804630000001 podStartE2EDuration="1.250680463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:57.249278102 +0000 UTC m=+3811.357317139" watchObservedRunningTime="2026-10-30 06:35:57.250680463 +0000 UTC m=+3811.358719492" Oct 30 06:35:57.703509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:57.703482 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:58.709366 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:58.709334 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:59.210874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:59.210844 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:35:59.211660 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:35:59.211642 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:36:01.426918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.426884 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6"] Oct 30 06:36:01.430453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.430425 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.446759 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.446737 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ee1c63\"" Oct 30 06:36:01.460179 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.460161 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-72d649bc-dockercfg-7pqlq\"" Oct 30 06:36:01.461342 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.461324 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6"] Oct 30 06:36:01.575042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.575011 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfbv\" (UniqueName: \"kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv\") pod \"payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.575238 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.575053 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.675932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.675898 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfbv\" (UniqueName: \"kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv\") pod \"payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.675932 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.675936 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.676598 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.676580 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.700963 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.700903 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfbv\" (UniqueName: \"kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv\") pod \"payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:01.744232 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:01.744198 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:02.053558 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:02.053533 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6"] Oct 30 06:36:02.055022 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:36:02.054995 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4d8f83_bc1d_4ceb_92e5_fed7b5f60529.slice/crio-2abcc0220f51b72ff2f1aa1d8658623b92328af3912b3761be76f7f4e955fd2e WatchSource:0}: Error finding container 2abcc0220f51b72ff2f1aa1d8658623b92328af3912b3761be76f7f4e955fd2e: Status 404 returned error can't find the container with id 2abcc0220f51b72ff2f1aa1d8658623b92328af3912b3761be76f7f4e955fd2e Oct 30 06:36:02.225874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:02.225837 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" event={"ID":"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529","Type":"ContainerStarted","Data":"2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960"} Oct 30 06:36:02.226032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:02.225888 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" event={"ID":"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529","Type":"ContainerStarted","Data":"2abcc0220f51b72ff2f1aa1d8658623b92328af3912b3761be76f7f4e955fd2e"} Oct 30 06:36:02.226032 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:02.225909 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:02.268580 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:02.268538 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" podStartSLOduration=1.268526232 podStartE2EDuration="1.268526232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:02.268515359 +0000 UTC m=+3816.376554391" watchObservedRunningTime="2026-10-30 06:36:02.268526232 +0000 UTC m=+3816.376565262" Oct 30 06:36:13.087045 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:36:13.087002 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:36:13.233364 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:13.233339 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:36:14.212275 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.212243 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:36:14.212732 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.212291 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 06:36:14.284800 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.284763 5980 generic.go:356] "Generic (PLEG): container finished" podID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerID="8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2" exitCode=0 Oct 30 06:36:14.284958 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.284833 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" event={"ID":"351ac9ed-73a7-49cc-b4f8-acaff83228eb","Type":"ContainerDied","Data":"8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2"} Oct 30 06:36:14.285187 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.285172 5980 scope.go:122] "RemoveContainer" containerID="8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2" Oct 30 06:36:14.286213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.286184 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:14.286325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.286252 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:14.286325 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:14.286284 5980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:15.292037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:15.291948 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" event={"ID":"351ac9ed-73a7-49cc-b4f8-acaff83228eb","Type":"ContainerStarted","Data":"17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff"} Oct 30 06:36:15.703590 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:15.703562 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:36:15.703834 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:15.703725 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:36:15.703957 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:15.703940 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 06:36:16.704225 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:16.704192 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:36:16.704631 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:16.704243 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 06:36:17.703709 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:17.703673 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:36:17.703892 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:17.703722 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 06:36:18.703706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:18.703588 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:36:18.703706 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:18.703651 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 06:36:19.716683 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:19.716652 5980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 06:36:19.717314 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:19.717281 5980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 06:36:20.818655 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:20.818625 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:36:20.819188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:20.819009 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:36:20.820140 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:20.820087 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:36:43.086537 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:36:43.086494 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:36:57.613147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:36:57.613101 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6"] Oct 30 06:37:02.613965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:02.613915 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerName="payload-pre-processing" containerID="cri-o://2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960" gracePeriod=25 Oct 30 06:37:03.232714 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:03.232683 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:37:07.624168 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:07.624084 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq"] Oct 30 06:37:07.624613 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:07.624352 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" containerID="cri-o://17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff" gracePeriod=30 Oct 30 06:37:12.884276 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:12.884251 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:37:13.051386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051320 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051351 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051386 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051381 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051411 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051428 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051457 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051497 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051602 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051641 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:37:13.051666 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051663 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy\") pod \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\" (UID: \"351ac9ed-73a7-49cc-b4f8-acaff83228eb\") " Oct 30 06:37:13.052040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051731 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:37:13.052040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051814 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:37:13.052040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.051842 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:37:13.052040 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.052035 5980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.052283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.052055 5980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-data\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.052283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.052069 5980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-workload-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.052283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.052083 5980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-credential-socket\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.052420 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.052325 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:13.054422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.054394 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:37:13.054533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.054458 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:37:13.054533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.054489 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "kube-api-access-wpgp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:13.054533 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.054511 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token" pod "351ac9ed-73a7-49cc-b4f8-acaff83228eb" (UID: "351ac9ed-73a7-49cc-b4f8-acaff83228eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:13.085887 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:37:13.085857 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:37:13.153546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.153518 5980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-token\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.153546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.153539 5980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-envoy\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.153546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.153548 5980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istio-podinfo\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.153752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.153559 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgp4\" (UniqueName: \"kubernetes.io/projected/351ac9ed-73a7-49cc-b4f8-acaff83228eb-kube-api-access-wpgp4\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.153752 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.153575 5980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351ac9ed-73a7-49cc-b4f8-acaff83228eb-istiod-ca-cert\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:13.232291 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.232262 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:37:13.549965 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.549933 5980 generic.go:356] "Generic (PLEG): container finished" podID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerID="17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff" exitCode=0 Oct 30 06:37:13.550142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.550015 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" Oct 30 06:37:13.550142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.550020 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" event={"ID":"351ac9ed-73a7-49cc-b4f8-acaff83228eb","Type":"ContainerDied","Data":"17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff"} Oct 30 06:37:13.550142 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.550129 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq" event={"ID":"351ac9ed-73a7-49cc-b4f8-acaff83228eb","Type":"ContainerDied","Data":"c25f0c80629722279a0cfa2057b88e33993da69e3485fa5ed43f1c4d2ce3854e"} Oct 30 06:37:13.550304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.550155 5980 scope.go:122] "RemoveContainer" containerID="17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff" Oct 30 06:37:13.559301 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.559282 5980 scope.go:122] "RemoveContainer" containerID="8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2" Oct 30 06:37:13.569837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.569817 5980 scope.go:122] "RemoveContainer" containerID="17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff" Oct 30 06:37:13.570070 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:37:13.570051 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff\": container with ID starting with 17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff not found: ID does not exist" containerID="17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff" Oct 30 06:37:13.570191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.570077 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff"} err="failed to get container status \"17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff\": rpc error: code = NotFound desc = could not find container \"17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff\": container with ID starting with 17d1e2b640622a2ed2f49bdf267f0fb461df83635d2f3934384ce370947385ff not found: ID does not exist" Oct 30 06:37:13.570191 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.570096 5980 scope.go:122] "RemoveContainer" containerID="8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2" Oct 30 06:37:13.570361 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:37:13.570341 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2\": container with ID starting with 8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2 not found: ID does not exist" containerID="8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2" Oct 30 06:37:13.570422 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.570367 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2"} err="failed to get container status \"8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2\": rpc error: code = NotFound desc = could not find container \"8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2\": container with ID starting with 8005e6da8f05b38060212c38d1dda47ef5a75f4286222b785c6e21c759c2e1e2 not found: ID does not exist" Oct 30 06:37:13.606388 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.606364 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq"] Oct 30 06:37:13.619653 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:13.619629 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ee1c63-openshift-default-579d85c845dwtpq"] Oct 30 06:37:14.011740 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:14.011707 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" path="/var/lib/kubelet/pods/351ac9ed-73a7-49cc-b4f8-acaff83228eb/volumes" Oct 30 06:37:22.854771 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.854745 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:37:22.890025 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.889960 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv\" (UniqueName: \"kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv\") pod \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " Oct 30 06:37:22.890186 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.890024 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume\") pod \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\" (UID: \"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529\") " Oct 30 06:37:22.890346 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.890323 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume" pod "2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" (UID: "2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:22.892188 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.892163 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv" pod "2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" (UID: "2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529"). InnerVolumeSpecName "kube-api-access-qpfbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:22.990881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.990852 5980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-plugins-config-volume\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:22.990881 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:22.990878 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfbv\" (UniqueName: \"kubernetes.io/projected/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529-kube-api-access-qpfbv\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:37:23.604598 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.604562 5980 generic.go:356] "Generic (PLEG): container finished" podID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerID="2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960" exitCode=0 Oct 30 06:37:23.604805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.604603 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" Oct 30 06:37:23.604805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.604652 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" event={"ID":"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529","Type":"ContainerDied","Data":"2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960"} Oct 30 06:37:23.604805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.604706 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6" event={"ID":"2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529","Type":"ContainerDied","Data":"2abcc0220f51b72ff2f1aa1d8658623b92328af3912b3761be76f7f4e955fd2e"} Oct 30 06:37:23.604805 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.604735 5980 scope.go:122] "RemoveContainer" containerID="2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960" Oct 30 06:37:23.613837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.613818 5980 scope.go:122] "RemoveContainer" containerID="2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960" Oct 30 06:37:23.614131 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:37:23.614090 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960\": container with ID starting with 2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960 not found: ID does not exist" containerID="2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960" Oct 30 06:37:23.614210 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.614150 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960"} err="failed to get container status \"2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960\": rpc error: code = NotFound desc = could not find container \"2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960\": container with ID starting with 2b8e5f3d37fc29aab21662c1bf648cf904c7735abeec8f4b755ca27096a5a960 not found: ID does not exist" Oct 30 06:37:23.658852 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.658827 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6"] Oct 30 06:37:23.672509 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:23.672488 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ee1c63-7df6dfmwv6"] Oct 30 06:37:24.011457 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:24.011383 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" path="/var/lib/kubelet/pods/2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529/volumes" Oct 30 06:37:28.116290 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.116253 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t"] Oct 30 06:37:28.117397 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117369 5980 cpu_manager.go:414] "containerMap: removing container" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerName="payload-pre-processing" Oct 30 06:37:28.117523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117399 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerName="payload-pre-processing" Oct 30 06:37:28.117523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117418 5980 cpu_manager.go:414] "containerMap: removing container" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" Oct 30 06:37:28.117523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117426 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" Oct 30 06:37:28.117696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117587 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4d8f83-bc1d-4ceb-92e5-fed7b5f60529" containerName="payload-pre-processing" Oct 30 06:37:28.117696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117603 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" Oct 30 06:37:28.117696 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.117614 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" Oct 30 06:37:28.122544 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.122525 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.153973 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.153945 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d358d1fd-kserve-self-signed-certs\"" Oct 30 06:37:28.155300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.155280 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t"] Oct 30 06:37:28.234444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.234411 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.234444 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.234448 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdt72\" (UniqueName: \"kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.234625 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.234480 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.234625 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.234591 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.234707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.234664 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.234707 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.234682 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.335781 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.335745 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.335948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.335789 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdt72\" (UniqueName: \"kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.335948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.335826 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.335948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.335853 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.335948 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.335905 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.336205 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.336026 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.336353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.336330 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.336353 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.336351 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.336523 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.336425 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.338243 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.338224 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.338491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.338474 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.360451 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.360430 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdt72\" (UniqueName: \"kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72\") pod \"e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.437541 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.437468 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:28.743791 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:28.743767 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t"] Oct 30 06:37:28.745974 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:37:28.745942 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode335854f_f32f_4337_8b4a_ef21a0da4abf.slice/crio-8bc9c7743cc305eefe271af78841181d80d3fa755a609bff47a1c8216092ff8a WatchSource:0}: Error finding container 8bc9c7743cc305eefe271af78841181d80d3fa755a609bff47a1c8216092ff8a: Status 404 returned error can't find the container with id 8bc9c7743cc305eefe271af78841181d80d3fa755a609bff47a1c8216092ff8a Oct 30 06:37:29.634895 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:29.634855 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" event={"ID":"e335854f-f32f-4337-8b4a-ef21a0da4abf","Type":"ContainerStarted","Data":"24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5"} Oct 30 06:37:29.635351 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:29.634910 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" event={"ID":"e335854f-f32f-4337-8b4a-ef21a0da4abf","Type":"ContainerStarted","Data":"8bc9c7743cc305eefe271af78841181d80d3fa755a609bff47a1c8216092ff8a"} Oct 30 06:37:29.678609 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:29.678558 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" podStartSLOduration=1.678544339 podStartE2EDuration="1.678544339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:37:29.678095579 +0000 UTC m=+3903.786134626" watchObservedRunningTime="2026-10-30 06:37:29.678544339 +0000 UTC m=+3903.786583372" Oct 30 06:37:30.437654 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:30.437617 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:34.289131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:34.288226 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-w8wfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:37:34.289131 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:34.288322 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-w8wfj" podUID="64aa156c-2329-46af-949b-482aa2562b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:37:38.437918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:38.437882 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:38.450808 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:38.450782 5980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:38.688390 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:38.688324 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:37:43.087222 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:37:43.087184 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:37:59.012412 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:37:59.012376 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t"] Oct 30 06:38:13.087651 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:38:13.087613 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:38:14.033361 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.033317 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" podUID="e335854f-f32f-4337-8b4a-ef21a0da4abf" containerName="main" containerID="cri-o://24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5" gracePeriod=45 Oct 30 06:38:14.286600 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.286531 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:38:14.397810 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.397771 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72\" (UniqueName: \"kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72\") pod \"e335854f-f32f-4337-8b4a-ef21a0da4abf\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " Oct 30 06:38:14.398026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.397879 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home\") pod \"e335854f-f32f-4337-8b4a-ef21a0da4abf\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " Oct 30 06:38:14.398026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.397917 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm\") pod \"e335854f-f32f-4337-8b4a-ef21a0da4abf\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " Oct 30 06:38:14.398026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.397958 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache\") pod \"e335854f-f32f-4337-8b4a-ef21a0da4abf\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " Oct 30 06:38:14.398026 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.397985 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs\") pod \"e335854f-f32f-4337-8b4a-ef21a0da4abf\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " Oct 30 06:38:14.398300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.398045 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir\") pod \"e335854f-f32f-4337-8b4a-ef21a0da4abf\" (UID: \"e335854f-f32f-4337-8b4a-ef21a0da4abf\") " Oct 30 06:38:14.398300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.398193 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home" pod "e335854f-f32f-4337-8b4a-ef21a0da4abf" (UID: "e335854f-f32f-4337-8b4a-ef21a0da4abf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:38:14.398300 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.398260 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache" pod "e335854f-f32f-4337-8b4a-ef21a0da4abf" (UID: "e335854f-f32f-4337-8b4a-ef21a0da4abf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:38:14.398475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.398438 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir" pod "e335854f-f32f-4337-8b4a-ef21a0da4abf" (UID: "e335854f-f32f-4337-8b4a-ef21a0da4abf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:38:14.398475 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.398460 5980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-home\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:14.398589 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.398482 5980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-model-cache\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:14.400416 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.400385 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72" pod "e335854f-f32f-4337-8b4a-ef21a0da4abf" (UID: "e335854f-f32f-4337-8b4a-ef21a0da4abf"). InnerVolumeSpecName "kube-api-access-hdt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:14.400531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.400426 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs" pod "e335854f-f32f-4337-8b4a-ef21a0da4abf" (UID: "e335854f-f32f-4337-8b4a-ef21a0da4abf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:14.400531 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.400486 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm" pod "e335854f-f32f-4337-8b4a-ef21a0da4abf" (UID: "e335854f-f32f-4337-8b4a-ef21a0da4abf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:38:14.499891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.499846 5980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-dshm\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:14.499891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.499876 5980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e335854f-f32f-4337-8b4a-ef21a0da4abf-tls-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:14.499891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.499886 5980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e335854f-f32f-4337-8b4a-ef21a0da4abf-tmp-dir\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:14.499891 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.499895 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdt72\" (UniqueName: \"kubernetes.io/projected/e335854f-f32f-4337-8b4a-ef21a0da4abf-kube-api-access-hdt72\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:14.879897 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.879863 5980 generic.go:356] "Generic (PLEG): container finished" podID="e335854f-f32f-4337-8b4a-ef21a0da4abf" containerID="24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5" exitCode=0 Oct 30 06:38:14.880228 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.879945 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" Oct 30 06:38:14.881002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.879954 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" event={"ID":"e335854f-f32f-4337-8b4a-ef21a0da4abf","Type":"ContainerDied","Data":"24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5"} Oct 30 06:38:14.881002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.880495 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t" event={"ID":"e335854f-f32f-4337-8b4a-ef21a0da4abf","Type":"ContainerDied","Data":"8bc9c7743cc305eefe271af78841181d80d3fa755a609bff47a1c8216092ff8a"} Oct 30 06:38:14.881002 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.880530 5980 scope.go:122] "RemoveContainer" containerID="24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5" Oct 30 06:38:14.893058 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.893031 5980 scope.go:122] "RemoveContainer" containerID="24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5" Oct 30 06:38:14.893503 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:38:14.893477 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5\": container with ID starting with 24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5 not found: ID does not exist" containerID="24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5" Oct 30 06:38:14.893665 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.893517 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5"} err="failed to get container status \"24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5\": rpc error: code = NotFound desc = could not find container \"24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5\": container with ID starting with 24349c10552ad5d5cfb66645e47f43696400983dbe40fa52013bdafa9c0dbdf5 not found: ID does not exist" Oct 30 06:38:14.936149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.936096 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t"] Oct 30 06:38:14.949781 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:14.949749 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d358d1fd-kserve-7556cd47fc-p6q8t"] Oct 30 06:38:16.032546 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:16.032502 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e335854f-f32f-4337-8b4a-ef21a0da4abf" path="/var/lib/kubelet/pods/e335854f-f32f-4337-8b4a-ef21a0da4abf/volumes" Oct 30 06:38:38.887739 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.887656 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-f87h8"] Oct 30 06:38:38.888149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.888003 5980 cpu_manager.go:414] "containerMap: removing container" podUID="e335854f-f32f-4337-8b4a-ef21a0da4abf" containerName="main" Oct 30 06:38:38.888149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.888014 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e335854f-f32f-4337-8b4a-ef21a0da4abf" containerName="main" Oct 30 06:38:38.888149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.888064 5980 cpu_manager.go:414] "containerMap: removing container" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" Oct 30 06:38:38.888149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.888069 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351ac9ed-73a7-49cc-b4f8-acaff83228eb" containerName="istio-proxy" Oct 30 06:38:38.888149 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.888150 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e335854f-f32f-4337-8b4a-ef21a0da4abf" containerName="main" Oct 30 06:38:38.891483 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.891458 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:38.901947 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.901919 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d424d79-2cb0-4849-abec-55e7bcecc77a-openshift-service-ca\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:38.902094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.902014 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lttxd\" (UniqueName: \"kubernetes.io/projected/2d424d79-2cb0-4849-abec-55e7bcecc77a-kube-api-access-lttxd\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:38.902094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.902044 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d424d79-2cb0-4849-abec-55e7bcecc77a-metrics-certs\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:38.902094 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.902078 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d424d79-2cb0-4849-abec-55e7bcecc77a-maas-api-tls\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:38.926252 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:38.926202 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-f87h8"] Oct 30 06:38:39.002687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.002653 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lttxd\" (UniqueName: \"kubernetes.io/projected/2d424d79-2cb0-4849-abec-55e7bcecc77a-kube-api-access-lttxd\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.002687 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.002687 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d424d79-2cb0-4849-abec-55e7bcecc77a-metrics-certs\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.002953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.002718 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d424d79-2cb0-4849-abec-55e7bcecc77a-maas-api-tls\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.002953 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.002879 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d424d79-2cb0-4849-abec-55e7bcecc77a-openshift-service-ca\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.003650 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.003554 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d424d79-2cb0-4849-abec-55e7bcecc77a-openshift-service-ca\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.005393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.005371 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d424d79-2cb0-4849-abec-55e7bcecc77a-metrics-certs\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.005825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.005803 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d424d79-2cb0-4849-abec-55e7bcecc77a-maas-api-tls\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.027491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.027458 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lttxd\" (UniqueName: \"kubernetes.io/projected/2d424d79-2cb0-4849-abec-55e7bcecc77a-kube-api-access-lttxd\") pod \"maas-api-595cc648c-f87h8\" (UID: \"2d424d79-2cb0-4849-abec-55e7bcecc77a\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.204887 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.204788 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:39.512056 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.512032 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-f87h8"] Oct 30 06:38:39.514825 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:38:39.514796 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d424d79_2cb0_4849_abec_55e7bcecc77a.slice/crio-1a2f78628e04d8e6f89e154e82fed32280dc66267267b959f6f3ee4ec25628b2 WatchSource:0}: Error finding container 1a2f78628e04d8e6f89e154e82fed32280dc66267267b959f6f3ee4ec25628b2: Status 404 returned error can't find the container with id 1a2f78628e04d8e6f89e154e82fed32280dc66267267b959f6f3ee4ec25628b2 Oct 30 06:38:39.516129 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:39.516095 5980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:38:40.013676 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:40.013640 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" event={"ID":"2d424d79-2cb0-4849-abec-55e7bcecc77a","Type":"ContainerStarted","Data":"ef90359bd508735a20ec53623cbbf53dc811a75fa163207992a300e21de61ffc"} Oct 30 06:38:40.014073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:40.013692 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" event={"ID":"2d424d79-2cb0-4849-abec-55e7bcecc77a","Type":"ContainerStarted","Data":"1a2f78628e04d8e6f89e154e82fed32280dc66267267b959f6f3ee4ec25628b2"} Oct 30 06:38:40.014073 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:40.013794 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:40.058725 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:40.058671 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" podStartSLOduration=1.6661655309999999 podStartE2EDuration="2.058656166s" totalImagesPullingTime="392.490635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:40.058167152 +0000 UTC m=+3974.166206214" watchObservedRunningTime="2026-10-30 06:38:40.058656166 +0000 UTC m=+3974.166695198" Oct 30 06:38:43.087019 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:38:43.086976 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:38:46.024215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:46.024182 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-f87h8" Oct 30 06:38:55.152567 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:55.152531 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-skq9m"] Oct 30 06:38:55.153009 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:55.152766 5980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" podUID="5eda9750-e242-40c3-984c-437d32d41f47" containerName="maas-api" containerID="cri-o://a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e" gracePeriod=30 Oct 30 06:38:56.452218 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.452178 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 06:38:56.512519 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.512430 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs\") pod \"5eda9750-e242-40c3-984c-437d32d41f47\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " Oct 30 06:38:56.512710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.512577 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca\") pod \"5eda9750-e242-40c3-984c-437d32d41f47\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " Oct 30 06:38:56.512710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.512622 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls\") pod \"5eda9750-e242-40c3-984c-437d32d41f47\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " Oct 30 06:38:56.512710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.512686 5980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc\" (UniqueName: \"kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc\") pod \"5eda9750-e242-40c3-984c-437d32d41f47\" (UID: \"5eda9750-e242-40c3-984c-437d32d41f47\") " Oct 30 06:38:56.513009 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.512979 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca" pod "5eda9750-e242-40c3-984c-437d32d41f47" (UID: "5eda9750-e242-40c3-984c-437d32d41f47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:56.515950 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.515918 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc" pod "5eda9750-e242-40c3-984c-437d32d41f47" (UID: "5eda9750-e242-40c3-984c-437d32d41f47"). InnerVolumeSpecName "kube-api-access-9tscc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:56.520576 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.520541 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs" pod "5eda9750-e242-40c3-984c-437d32d41f47" (UID: "5eda9750-e242-40c3-984c-437d32d41f47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:56.520851 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.520812 5980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls" pod "5eda9750-e242-40c3-984c-437d32d41f47" (UID: "5eda9750-e242-40c3-984c-437d32d41f47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:56.613837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.613791 5980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda9750-e242-40c3-984c-437d32d41f47-openshift-service-ca\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.613837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.613834 5980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-maas-api-tls\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.614162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.613850 5980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tscc\" (UniqueName: \"kubernetes.io/projected/5eda9750-e242-40c3-984c-437d32d41f47-kube-api-access-9tscc\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:56.614162 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:56.613865 5980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eda9750-e242-40c3-984c-437d32d41f47-metrics-certs\") on node \"ip-10-0-138-15.ec2.internal\" DevicePath \"\"" Oct 30 06:38:57.118133 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.110496 5980 generic.go:356] "Generic (PLEG): container finished" podID="5eda9750-e242-40c3-984c-437d32d41f47" containerID="a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e" exitCode=0 Oct 30 06:38:57.118488 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.118455 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" event={"ID":"5eda9750-e242-40c3-984c-437d32d41f47","Type":"ContainerDied","Data":"a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e"} Oct 30 06:38:57.118689 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.118671 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" event={"ID":"5eda9750-e242-40c3-984c-437d32d41f47","Type":"ContainerDied","Data":"c1207101f40727a48e591c9974811f10e33f019574fb77a6baf61d9c6574b726"} Oct 30 06:38:57.118825 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.118810 5980 scope.go:122] "RemoveContainer" containerID="a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e" Oct 30 06:38:57.119148 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.119130 5980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-skq9m" Oct 30 06:38:57.147806 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.147515 5980 scope.go:122] "RemoveContainer" containerID="a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e" Oct 30 06:38:57.148197 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:38:57.148073 5980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e\": container with ID starting with a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e not found: ID does not exist" containerID="a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e" Oct 30 06:38:57.148197 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.148133 5980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e"} err="failed to get container status \"a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e\": rpc error: code = NotFound desc = could not find container \"a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e\": container with ID starting with a831ed36d8052104f94936ac2a894226ae12e2dd6ae719759cf5bd30c31de36e not found: ID does not exist" Oct 30 06:38:57.198735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.198556 5980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-skq9m"] Oct 30 06:38:57.219394 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:57.219360 5980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-skq9m"] Oct 30 06:38:58.032135 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:38:58.028733 5980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eda9750-e242-40c3-984c-437d32d41f47" path="/var/lib/kubelet/pods/5eda9750-e242-40c3-984c-437d32d41f47/volumes" Oct 30 06:39:13.188142 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:39:13.180365 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:39:14.028710 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:14.028648 5980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-8r2sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:39:14.028961 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:14.028729 5980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-8r2sh" podUID="ece567c8-3dcb-4fd2-9ff3-48da693b4b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:39:37.265433 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:37.265400 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784db4fd-vzlfm_dd876cb0-6d24-4d4f-9a21-65cf65d92188/authorino/0.log" Oct 30 06:39:43.086850 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:39:43.086812 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:39:49.928269 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:49.928235 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-g8rjp_9253b348-62ae-4bf7-b6fe-8b25d841ec93/copy-manifests/0.log" Oct 30 06:39:49.976981 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:49.976947 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-g8rjp_9253b348-62ae-4bf7-b6fe-8b25d841ec93/manager/0.log" Oct 30 06:39:51.225837 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:51.225810 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-f87h8_2d424d79-2cb0-4849-abec-55e7bcecc77a/maas-api/0.log" Oct 30 06:39:53.273691 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:53.273664 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p64x4_b0f53d1a-6fce-4313-b6b9-3153e29f7f13/postgres/0.log" Oct 30 06:39:54.787819 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:54.787788 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784db4fd-vzlfm_dd876cb0-6d24-4d4f-9a21-65cf65d92188/authorino/0.log" Oct 30 06:39:55.421491 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:55.421464 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vfcq8_27863d34-3115-428c-a857-077f149b18ae/kuadrant-console-plugin/0.log" Oct 30 06:39:55.641332 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:55.641293 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-rjtn7_6b072765-4f9f-4379-b544-ecc9650203c3/manager/0.log" Oct 30 06:39:57.535888 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:57.535840 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8596b47b87-pk7dd_ef9b6de8-58d0-4fba-9193-a23327d0d792/kube-auth-proxy/0.log" Oct 30 06:39:57.904506 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:57.904469 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-74tmp_499c0ecb-e7fc-47e7-9435-9be34a40ecb3/istio-proxy/0.log" Oct 30 06:39:58.333022 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:58.332987 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-8r2sh_ece567c8-3dcb-4fd2-9ff3-48da693b4b1e/istio-proxy/0.log" Oct 30 06:39:59.531373 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:59.531340 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-w8wfj_64aa156c-2329-46af-949b-482aa2562b42/istio-proxy/0.log" Oct 30 06:39:59.716215 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:39:59.716190 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dlwvz_5a0e5f24-1add-4aaf-ae9d-7bab12f8487b/payload-pre-processing/0.log" Oct 30 06:40:01.028485 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:01.028453 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zx676_2f109c85-5be4-41f3-b199-0d6da6a35a82/main/0.log" Oct 30 06:40:01.812700 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:01.812670 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj_3c016552-431e-4686-af6b-983057f85c80/main/0.log" Oct 30 06:40:02.222892 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:02.222855 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5_206d662f-b8a8-48b5-b8f9-c74d9785870f/main/0.log" Oct 30 06:40:03.156320 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:03.156291 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784db4fd-vzlfm_dd876cb0-6d24-4d4f-9a21-65cf65d92188/authorino/0.log" Oct 30 06:40:13.086299 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:40:13.086213 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:40:15.505883 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:15.505855 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-g8rjp_9253b348-62ae-4bf7-b6fe-8b25d841ec93/copy-manifests/0.log" Oct 30 06:40:15.545471 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:15.545436 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-g8rjp_9253b348-62ae-4bf7-b6fe-8b25d841ec93/manager/0.log" Oct 30 06:40:16.484914 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:16.484861 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-f87h8_2d424d79-2cb0-4849-abec-55e7bcecc77a/maas-api/0.log" Oct 30 06:40:18.492794 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:18.492750 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-p64x4_b0f53d1a-6fce-4313-b6b9-3153e29f7f13/postgres/0.log" Oct 30 06:40:19.616538 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:19.616500 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784db4fd-vzlfm_dd876cb0-6d24-4d4f-9a21-65cf65d92188/authorino/0.log" Oct 30 06:40:20.231770 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:20.231745 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vfcq8_27863d34-3115-428c-a857-077f149b18ae/kuadrant-console-plugin/0.log" Oct 30 06:40:20.417454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:20.417406 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-rjtn7_6b072765-4f9f-4379-b544-ecc9650203c3/manager/0.log" Oct 30 06:40:21.993564 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:21.993529 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8596b47b87-pk7dd_ef9b6de8-58d0-4fba-9193-a23327d0d792/kube-auth-proxy/0.log" Oct 30 06:40:22.362918 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:22.362882 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-74tmp_499c0ecb-e7fc-47e7-9435-9be34a40ecb3/istio-proxy/0.log" Oct 30 06:40:22.773903 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:22.773816 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-8r2sh_ece567c8-3dcb-4fd2-9ff3-48da693b4b1e/istio-proxy/0.log" Oct 30 06:40:23.963688 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:23.963656 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-w8wfj_64aa156c-2329-46af-949b-482aa2562b42/istio-proxy/0.log" Oct 30 06:40:24.146012 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:24.145977 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dlwvz_5a0e5f24-1add-4aaf-ae9d-7bab12f8487b/payload-pre-processing/0.log" Oct 30 06:40:25.076393 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:25.076361 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zx676_2f109c85-5be4-41f3-b199-0d6da6a35a82/main/0.log" Oct 30 06:40:25.851453 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:25.851421 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b827vfj_3c016552-431e-4686-af6b-983057f85c80/main/0.log" Oct 30 06:40:26.242787 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:40:26.242696 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fspr5_206d662f-b8a8-48b5-b8f9-c74d9785870f/main/0.log" Oct 30 06:40:43.090207 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:40:43.090163 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:41:13.086566 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:41:13.086523 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:41:42.192802 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:42.192708 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wb5rz_20949c65-577f-4bef-9494-b8ddec17c982/global-pull-secret-syncer/0.log" Oct 30 06:41:43.058617 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:43.058590 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sxrx6_7b010236-561c-4748-842a-648d3290ff2b/konnectivity-agent/0.log" Oct 30 06:41:43.086582 ip-10-0-138-15 kubenswrapper[5980]: E1030 06:41:43.086546 5980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ctntz" containerName="extractor" Oct 30 06:41:43.433577 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:43.433544 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-15.ec2.internal_eddf44eee240eccf9696b703c5918959/haproxy/0.log" Oct 30 06:41:50.173223 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:50.173186 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54784db4fd-vzlfm_dd876cb0-6d24-4d4f-9a21-65cf65d92188/authorino/0.log" Oct 30 06:41:50.506455 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:50.506378 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vfcq8_27863d34-3115-428c-a857-077f149b18ae/kuadrant-console-plugin/0.log" Oct 30 06:41:50.676571 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:50.676532 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-rjtn7_6b072765-4f9f-4379-b544-ecc9650203c3/manager/0.log" Oct 30 06:41:51.789260 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.789215 5980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq"] Oct 30 06:41:51.789699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.789647 5980 cpu_manager.go:414] "containerMap: removing container" podUID="5eda9750-e242-40c3-984c-437d32d41f47" containerName="maas-api" Oct 30 06:41:51.789699 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.789662 5980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda9750-e242-40c3-984c-437d32d41f47" containerName="maas-api" Oct 30 06:41:51.789810 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.789766 5980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda9750-e242-40c3-984c-437d32d41f47" containerName="maas-api" Oct 30 06:41:51.792860 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.792843 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.811037 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.811015 5980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-88m2b\"/\"default-dockercfg-hccqc\"" Oct 30 06:41:51.824283 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.824259 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-88m2b\"/\"openshift-service-ca.crt\"" Oct 30 06:41:51.824455 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.824437 5980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-88m2b\"/\"kube-root-ca.crt\"" Oct 30 06:41:51.826463 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.826442 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq"] Oct 30 06:41:51.874866 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.874834 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-proc\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.875030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.874885 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-podres\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.875030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.874952 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-sys\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.875030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.874997 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnp47\" (UniqueName: \"kubernetes.io/projected/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-kube-api-access-mnp47\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.875030 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.875027 5980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-lib-modules\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976253 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976219 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-lib-modules\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976304 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-proc\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976341 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-podres\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976362 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-sys\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976385 5980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnp47\" (UniqueName: \"kubernetes.io/projected/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-kube-api-access-mnp47\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976391 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-lib-modules\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976423 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-proc\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976454 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976443 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-podres\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:51.976715 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:51.976444 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-sys\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:52.001042 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.001007 5980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnp47\" (UniqueName: \"kubernetes.io/projected/94f7a8f0-f27b-4c6b-b7fd-5c98364687dd-kube-api-access-mnp47\") pod \"perf-node-gather-daemonset-g6kvq\" (UID: \"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:52.105527 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.105488 5980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:52.408127 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.408081 5980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq"] Oct 30 06:41:52.410131 ip-10-0-138-15 kubenswrapper[5980]: W1030 06:41:52.410088 5980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod94f7a8f0_f27b_4c6b_b7fd_5c98364687dd.slice/crio-c1be8ffa14f4115e3b41d74dad458777709494f3d3c2d927e8e67b3d3697d230 WatchSource:0}: Error finding container c1be8ffa14f4115e3b41d74dad458777709494f3d3c2d927e8e67b3d3697d230: Status 404 returned error can't find the container with id c1be8ffa14f4115e3b41d74dad458777709494f3d3c2d927e8e67b3d3697d230 Oct 30 06:41:52.913722 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.913679 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" event={"ID":"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd","Type":"ContainerStarted","Data":"e1907e64192dfa50c8217c08139e99d167e3d260be3bccf8a782e11d6d0aa3c2"} Oct 30 06:41:52.914172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.913781 5980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" event={"ID":"94f7a8f0-f27b-4c6b-b7fd-5c98364687dd","Type":"ContainerStarted","Data":"c1be8ffa14f4115e3b41d74dad458777709494f3d3c2d927e8e67b3d3697d230"} Oct 30 06:41:52.914172 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.913809 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:41:52.957842 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:52.957794 5980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" podStartSLOduration=1.957781175 podStartE2EDuration="1.957781175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:52.956429347 +0000 UTC m=+4167.064468373" watchObservedRunningTime="2026-10-30 06:41:52.957781175 +0000 UTC m=+4167.065820208" Oct 30 06:41:53.415213 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:53.415181 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jhh2s_f8b282a1-edcf-42d2-9f53-c995efc488cd/kube-state-metrics/0.log" Oct 30 06:41:53.516310 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:53.516273 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jhh2s_f8b282a1-edcf-42d2-9f53-c995efc488cd/kube-rbac-proxy-main/0.log" Oct 30 06:41:53.634662 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:53.634628 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jhh2s_f8b282a1-edcf-42d2-9f53-c995efc488cd/kube-rbac-proxy-self/0.log" Oct 30 06:41:53.866612 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:53.866581 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-6qhbq_bff6f637-e30a-47b8-88fe-a5aa3dfd5e02/monitoring-plugin/0.log" Oct 30 06:41:53.982870 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:53.982845 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68zhv_714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/node-exporter/0.log" Oct 30 06:41:54.091637 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:54.091604 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68zhv_714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/kube-rbac-proxy/0.log" Oct 30 06:41:54.203833 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:54.203757 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-68zhv_714aff30-d1b2-4ed7-9a37-e7674c4c2ad3/init-textfile/0.log" Oct 30 06:41:55.030718 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.030681 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wpplk_98584416-9e1b-4358-8a58-23d4b123be72/kube-rbac-proxy-main/0.log" Oct 30 06:41:55.143190 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.143160 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wpplk_98584416-9e1b-4358-8a58-23d4b123be72/kube-rbac-proxy-self/0.log" Oct 30 06:41:55.253735 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.253709 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-wpplk_98584416-9e1b-4358-8a58-23d4b123be72/openshift-state-metrics/0.log" Oct 30 06:41:55.380452 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.380423 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/prometheus/0.log" Oct 30 06:41:55.488043 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.488015 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/config-reloader/0.log" Oct 30 06:41:55.597685 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.597654 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/thanos-sidecar/0.log" Oct 30 06:41:55.709763 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.709687 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/kube-rbac-proxy-web/0.log" Oct 30 06:41:55.838874 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.838850 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/kube-rbac-proxy/0.log" Oct 30 06:41:55.950304 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:55.950275 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/kube-rbac-proxy-thanos/0.log" Oct 30 06:41:56.096525 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:56.096494 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_10ea2de1-bf89-40da-a84b-2350d913bd66/init-config-reloader/0.log" Oct 30 06:41:58.928862 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:41:58.928834 5980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-g6kvq" Oct 30 06:42:05.482545 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:42:05.482514 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rm8hh_334ad7cd-4277-4572-b7e2-b0e2ececd5ff/dns/0.log" Oct 30 06:42:05.589566 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:42:05.589532 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rm8hh_334ad7cd-4277-4572-b7e2-b0e2ececd5ff/kube-rbac-proxy/0.log" Oct 30 06:42:05.948814 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:42:05.948788 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q2qpx_8885b33d-691c-4ba4-916d-c1e27fbd1cd3/dns-node-resolver/0.log" Oct 30 06:42:07.227147 ip-10-0-138-15 kubenswrapper[5980]: I1030 06:42:07.227095 5980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mh7z_f9766570-5a16-4a4a-aa4e-900bbbed459c/node-ca/0.log"